NAME

Queue::Q - Mix-and-match Queue Implementations and Backends

SYNOPSIS

# Pick you queue type and a back-end:

# Very primitive FIFO queue (abstract interface)
use Queue::Q::NaiveFIFO;
# ... and its pure-Perl, in-memory implementation
use Queue::Q::NaiveFIFO::Perl;
# ... its Redis-based implementation
use Queue::Q::NaiveFIFO::Redis;

# "Reliable" enqueue, claim, mark-as-done FIFO queue:
use Queue::Q::ClaimFIFO; # abstract interface
# ... and again, its pure-Perl, in-memory implementation
use Queue::Q::ClaimFIFO::Perl;
# ... and its Redis-based implementation
use Queue::Q::ClaimFIFO::Redis;

# A composite, distributed queue, built from shards of
# the above types with weak global and strict local
# ordering, scaling beyond single nodes.
use Queue::Q::DistFIFO; 

# Usage is somewhat similar for most queue types:
my $queue = Queue::Q::ClaimFIFO::Redis->new(
    server      => 'redis-01',
    port        => '6379',
    queue_name  => "image_queue"
);

$q->enqueue_item( $img_url );

# In another process on another machine:
while (defined( my $item = $q->claim_item )) {
    # Do some work!
    $q->mark_item_as_done($item);
}

# n-ary versions exist, too:
$q->enqueue_items( @img_urls );
my @items = $q->claim_items(10); # 10 at a time, padded with undef
$q->mark_items_as_done(@items);

DESCRIPTION

This is an experimental module. The interface may change without notice. Before using it in production, please get in touch with the authors!

Queue::Q is a collection of queue implementations each with multiple backends. Right now, it comes with three basic queues:

Queue::Q::NaiveFIFO

A naive FIFO queue NaiveFIFO. Supports enqueuing data and later claiming it in the order it was enqueued. Strict ordering, low-latency, high-throughput, no resilience against crashing workers.

Queue::Q::ClaimFIFO

A claim/mark-as-done FIFO queue ClaimFIFO. Supports enqueuing data (as Queue::Q::ClaimFIFO::Item objects), claiming items, and keeping track of items-being-worked-on until they are reported as done. Strict ordering, slightly higher latency and lower throughput than the naive FIFO queue. Resilience against crashing workers when combined with a clean-up script that checks for old, claimed items.

Queue::Q::ReliableFIFO

Similar interface to the ClaimFIFO queue type, but only comes with a Redis-backed implementation now. Much faster for the basic operations then ClaimFIFO since it doesn't always have to execute Lua scripts. See Queue::Q::ReliableFIFO for details.

The first two basic queues come with two back-end implementations each right now: One is a very simple, in-memory, single-process implementation Queue::Q::NaiveFIFO::Perl and Queue::Q::ClaimFIFO::Perl respectively. The other is an implementation based on Redis: Queue::Q::NaiveFIFO::Redis and Queue::Q::ClaimFIFO::Redis.

As noted above, the ReliableFIFO queue has a Redis-based backend only for now.

In addition to the basic queue types, the distribution contains Queue::Q::DistFIFO, an implementation of a distributed queue that can use the basic queues as shards (but only one type of shard per distributed queue). It supports all operations of the basic queues, but does not enforce strict global ordering, but weak global and strict local ordering. In an early test, two mid-range servers running multiple instances of Redis each sustained 800k-1M transactions per second with a naive-type distributed queue. DistFIFO has not been tested with the ReliableFIFO implementation as building blocks yet!

ACKNOWLEDGMENT

This module was originally developed for Booking.com. With approval from Booking.com, this module was generalized and put on CPAN, for which the authors would like to express their gratitude.

AUTHOR

Herald van der Breggen, <herald.vanderbreggen@booking.com>

Steffen Mueller, <smueller@cpan.org>

COPYRIGHT AND LICENSE

Copyright (C) 2012, 2013 by Steffen Mueller

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.1 or, at your option, any later version of Perl 5 you may have available.