NAME
Argon - Simple, fast, and flexible distributed computing
VERSION
version 0.18
DESCRIPTION
Argon is a distributed processing platform built for Perl. It is designed to offer a simple, flexible, system for quickly building scalable systems with the minimum impact on existing workflows and code structure.
QUICK START
An argon system is controlled by a manager process, whose job it is to schedule tasks among registered workers.
MANAGER
A manager process is started with ar-manager. The manager, workers, and clients must all use the same key (a file containing a key phrase used for encryption).
ar-manager --host localhost --port 8000 --key path/to/secret --verbose 7WORKER
Workers are started with ar-worker and must use the same key as the manager.
ar-worker --mgr mgrhost:8000 --capacity 8 --key path/to/secret --verbose 7CLIENT
Connecting to an Argon service is straightforward.
use Argon::Client;
use AnyEvent;
# Connect
my $cv = AnyEvent->condvar;
my $ar = Argon::Client->new(
host => 'mgrhost',
port => 8000,
keyfile => 'path/to/key',
ready => $cv,
retry => 1,
);
# Connected!
$cv->recv;A code ref (or any callable reference) may be passed using the ready parameter which will be called once the client is connected. The example uses a condition variable (see "CONDITION VARIABLES" in AnyEvent) to sleep until it is called, making the connection blocking.
RUNNING TASKS
Once connected, there are a number of ways to schedule tasks with the manager, the most basic being the "queue" in Argon::Client method.
$client->queue('My::Task::Class', $arg_list, sub {
my $reply = shift;
my $result = $reply->result;
});There are a couple things to note here. The task class is any class that defines both a new and a run method. The $arg_list will be passed to new. A subroutine is passed which is called when the result is ready. The call to the result method will croak if the task failed.
If retry was set when the client was connected, the task will retry on a logarithmic backoff until the server has the capacity to process the task (see "PREDICTABLE PERFORMANCE DEGREDATION" in Argon).
If the workers were started with the --allow-eval switch, the client may pass code references directly to be evaluated by the workers using the "process" in Argon::Client method.
local $Argon::ALLOW_EVAL = 1;
$client->process(sub { ... }, $arg_list, sub {
...
});PREDICTABLE PERFORMANCE DEGREDATION
One of the key problems with many task queue implementations is the manner in which the system recovers from an interruption in service. In most cases, tasks continue to pile up while the system is unavailable. Once the service is again ready to process tasks, a significant backlog has built up, creating further delay for new tasks added to the queue. This creates a log jam that is often accompanied by incidental service slowdowns that can be difficult to diagnose (for example, overloaded workers clearing out the backlog tie up the database, causing other services to slow down).
Argon prevents these cases by placing the responsibility for the backlog on the client. When the manager determines that the system has reached max capacity, new tasks are rejected until there is room in the queue. From the perspective of the client, there is still a delay in the processing of tasks, but the task queue itself never becomes overloaded and the performance degredation will never overflow onto neighborhing systems as a result.
Another adavantage of having a bounded queue is that clients are aware of the backlog and may report this to callers. System administrators may effectively plan for and respond to increased load by spinning up new servers as needed because they can reliably predict the performance of the system under load given a reliable estimate of the cost imposed by the tasks being performed.
AUTHOR
Jeff Ober <sysread@fastmail.fm>
COPYRIGHT AND LICENSE
This software is copyright (c) 2017 by Jeff Ober.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.