NAME

Vayne - Distribute task queue

SYNOPSIS

use Vayne;
use Vayne::Callback;
use Vayne::Tracker;

my @workload = <>; chomp @workload;
my $tracker = Vayne::Tracker->new();
my $step = Vayne->task('foo');
my $taskid = $tracker->add_task(
     'region:region-first',
     {
         name   => 'foo',
         step   => $step,
         expire => 90,
     },
     @workload
);
my $call = Vayne::Callback->new();
my $stat = $call->wait($taskid);

GETTING STARTED

# First time only
> vayne-init -d $HOME/vayne

# Setup configurations in $HOME/vayne/conf (zookeeper mongodb redis)

# Add our first region, then the region info will upload to zk server.
> vayne-ctrl --set --region region-first --server redisserver:port --password redispasswd

# Define our first task, $HOME/vayne/task/foo

  # check server ssh server
  - name: 'check ssh'                     #step's name
    worker: tcp                           #step's worker
    param:                                #step's parameters
      port: 22
      input: ''
      check: '^SSH-2.0-OpenSSH'

  - name: 'foo'
    worker: dump
    param:
      bar: baz

  - name: 'only suc'
    need:
      - 'check ssh': 1
    worker: dump
    param:
      - array
      - key1: value1
        key2: value2

  # tracke the job result
  - name: 'tracker'
    worker: track

# Switch the server you run workers to our first region.
> vayne-ctrl --switch --region region-first

# Run workers.
> $HOME/vayne/worker/tcp &
> $HOME/vayne/worker/dump &
> $HOME/vayne/worker/tracker &

# Run task tracker.
> vayne-tracker &

# Submit our task by CLI.
> echo '127.0.0.1'|vayne-task --add --name foo --expire 60 --strategy region:region-first
# or
> vayne-task --add --name foo --expire 60 --strategy region:region-first < server_list

# Query our task through taskid by CLI.
> vayne-task --taskid 9789F5E6-2644-11E6-A6F0-AF9AF8F9E07F --query

# Or use Vayne lib in your program like SYNOPSIS.

DESCRIPTION

Vayne is a distribute task queue with many feature.

FEATURE

Logical Region with Flexible Spawning Strategy: Has the concept of logical region. You can spawn task into different region with strategy. Spawning strategy can be easily write.
Custome Task Flow with Reusable Worker: Worker is a process can do some specific stuffs. Step have a step's name, a worker's name and parameters. You can define custome task flow by constructing any steps.
Simple Worker Interface with Good Performance: Vayne::Worker is using Coro which can provide excellent performance in network IO. Worker has a simple interface to write, also you can use Coro::* module to enhance worker performance. Whole system is combined with Message Queue. You can get a better performance easily by increasing the worker counts while MQ is not the bottleneck.

HOW IT WORKS

                                                                           +--------+
                                                                           |Worker A| x N
                                                                           +--------+
                                                     +------------+                        Workers may run on several servers
                                                     |            |        +--------+
                                                     |  Region A  |        |Worker B| x N
                                                     |            |        +--------+
                                                     +------------+
                                                                             .....
+-----------+                                                              +----------+
|           |                                        +------------+        |JobTracker| x N
| Task Conf |                                        |            |        +----------+
|           |                                        |  Region B  |
| +-------+ |                                        |            |
| | step1 | |    +-----------+                       +------------+                                                 +-----------+          +-----------+
| +-------+ |    |           |        Spawn Jobs                                           Save Job Information     |           | +------> |           |
|           | +  | workloads |    +---------------->                                       +--------------------->  |  Mongodb  |          |TaskTracker|
| +-------+ |    |           |       with Strategy                                          to Center Mongodb       |           | <------+ |           |
| | step2 | |    +-----------+                       +------------+                                                 +-----------+          +-----------+
| +-------+ |                                        |            |
|    ...    |                                        |  Region C  |          .....                                       ^
| +-------+ |                                        |            |                                                      |
| | stepN | |                                        +------------+                                                      |
| +-------+ |                                                                                                            |
|           |                                                                                                            |
+-----------+                                        +------------+                                                      |
                                                     |            |                                                      |
                                                     |  Region D  |                                                      |
              |                                      |            |                                                      |
              |                                      +------------+                                                      |
              |                                                                                                          |
              |                                                                                                          |
              |                                                                                                          |
              |                                                                                                          |
              |                                Save Task Information to Center Mongodb                                   |
              +----------------------------------------------------------------------------------------------------------+

0. Task Conf & Step

The Task Conf is combined with several ordered steps. Each step have a step's name, a worker's name and parameters. Workload will be prosessed step by step.

1. Spawn Task

Vayne support CLI and API to spawn a task. A task contain numbers of jobs. Task info will write to task collection in mongodb first. Jobs will be hashed into saperated region by strategy. Then enqueue jobs to their region's redis queue named by first step of the job, and write to job collection in mongodb.

2. Queue & Region

Like Redis::JobQueue, Vayne use redis for job queuing and job info caching. The data structure is nearly the same as "JobQueue data structure stored in Redis" in Redis::JobQueue.

Each region has a queue(redis server). Both their infomation are saved on zookeeper server.

Each real server which you want to run workers should belong to a region.

Worker will register its names under real server's region on zookeeper server when it start.

Details see "DATA STRUCTURE" in Vayne::Zk.

2. Worker

When it start, worker register its names on zookeeper server. Then generate some Coro threads below:

Check the Registration

Go die when the registration changed. Ex: Region info changed; Real Server switch to another region; Connection to zk failed.

* The worker will die very quickly when zookeeper server is not available. It may cause some problems. Should be careful.

Job Consumer

BLPOP queues which worker registered, then put the job into Coro::Channel

Worker

Get job from Coro::Channel, and do the stuff with it. Tag the result and status on the job. Put the job to update Coro::Channel.

Update Job Info

Get Job from update Coro::Channel. Push the job to next queue according to the job's step.

INT, TERM, HUP signals will be catched. Then graceful stop the worker.

4. JOB TRACKER

Job tracker is a special worker, it just send the job info dealed by previous workers to mongodb. Usually 'tracker' should be the last step of a job.

5. TASK TRACKER

Script vayne-tracker Loop bla..

BACKEND

Redis-3.2 http://redis.io/

Zookeeper-3.3.6 http://zookeeper.apache.org/

MongoDB-3.0.6 https://www.mongodb.com/

DATA STRUCTURE

Zookeeper

"DATA STRUCTURE" in Vayne::Zk

Redis

Data Structure for job&queue is nearly the same as "JobQueue data structure stored in Redis" in Redis::JobQueue

MongoDB

bla bla..

HOW TO WRITE A WORKER