/ 
App-Wubot-0.3.8
/ App::Wubot::LocalMessageStore

NAME

App::Wubot::LocalMessageStore - add or remove messages from a local wubot SQLite message queue

VERSION

version 0.3.8

SYNOPSIS

use App::Wubot::LocalMessageStore;

my $messenger = App::Wubot::LocalMessageStore->new();
my $directory = "/path/to/queue/directory";

$messenger->store( { %{ $message } }, $directory );

# scalar context, get the message, immediately deleting it from
# the queue
my $got_message = $messenger->get( $directory );

# array context, get the message and return a callback that can be
# called to delete the message after it has been successfully
# processed.
my ( $got_message, $callback ) = $messenger->get( $directory );
# do something here to process the message
# delete the message after processing
$callback->();

DESCRIPTION

Wubot uses LocalMessageStore to add messages to a FIFO queue implemented in SQLite for asynchronous processing.

Common uses of LocalMessageStore include the queue where wubot-monitor stores messages for reactions by wubot-reactor.

When a message in the queue has been processed, it will not be immediately removed from the queue. Instead, the 'seen' flag will be set to the timestamp when the message was marked as processed.

GUARANTEED MESSAGE PROCESSING

The callback mechanism (see the example above) is used to ensure that the message has been successfully processed before it is deleted. If the wubot process gets shut down in the middle of processing a message, the message will not be removed from the queue. This could sometimes lead to reacting to a message more than once, but ensures that the reaction will always occur.

SUBROUTINES/METHODS

initialize_db( $directory )

This will get called the first time a queue is accessed in a running process. A App::Wubot::SQLite object will be created for the queue.

When initializing the database connection, all items that were marked 'seen' more than 24 hours ago will be deleted from the queue, and the current number of messages remaining in the queue in the 'seen' and 'unseen' state will be logged. Since the wubot processes are automatically restarted every night just after midnight, this ensures that the queue will not get too long, since long queues can significantly degrade performance.

store( $message, $directory )

Given a message (a simple or multi-level hash), store it in the queue in the specified directory.

If the message does not already have a checksum, lastupdate, or hostname, then those fields will be determined.

The message will be serialized using YAML::XS, and the result will be stored in the 'data' column.

delete_seen( $directory, $age )

Delete all queue items where the 'seen' time is older than the specified 'age' in seconds.

Also calls the 'vacuum' command on the database to reclaim any unused space and improve performance.

get( $directory )

Get the oldest unseen item in the queue, deserialize the message, and return the message data structure.

If called in array context, will also return a callback which can be used to delete the message (i.e. mark it 'seen') after it has been processed.

get_counts( $directory )

Given the directory of a queue, return three statistics:

- number of messages in the queue marked 'seen'
- number of messsages in the queue left to be processed
- total number of messages in the queue
checksum( $message )

Given a message, calculate the md5sum of the message. This involves serializing the message with YAML::XS and then calculating the md5_hex of the serialized message. The generated 'checksum' field can be useful to detect duplicate messages.

YAML::XS

This library requires YAML::XS, as I have experienced issues serializing and/or deserializing messages with every other perl YAML library I have tried. YAML::XS has been extremely reliable; I have yet to encounter a message that it can not handle.