NAME

HTTP::LoadGen - a HTTP load generator toolset

SYNOPSIS

use HTTP::LoadGen;

use HTTP::LoadGen qw/:all/;

# ask import() to replace the built-in 'rand' operator by our
# thread-specific RNG (uses *CORE::GLOBAL::rand)
use HTTP::LoadGen qw/-rand :all/;

######################
# the load generator #
######################

HTTP::LoadGen::loadgen \%config;

#######################
# auxiliary functions #
#######################

# process management

# create a collection of worker processes
$handle=HTTP::LoadGen::create_proc $nproc, $inithnd, $handler, $exithnd;

# start main processing and wait for then to finish
%result=%{HTTP::LoadGen::start_proc $handle};

# thread management

# create a collection of threads
$sem=HTTP::LoadGen::ramp_up
    $procnr, $nproc, $start, $max, $duration, $handler;

# wait for them to finish
$sem->down;

# idle a bit
HTTP::LoadGen::delay $prefix, \%param;

# get current thread number
$nr=HTTP::LoadGen::threadnr;

# get the configuration hash
$config=HTTP::LoadGen::options;

# get/set thread-specific user data
$data=HTTP::LoadGen::userdata;
HTTP::LoadGen::userdata=$data;

# get/set thread specific random number generator
$rng=HTTP::LoadGen::rng;
HTTP::LoadGen::rng=$rng;

# next random number
$random=HTTP::LoadGen::rnd $max;

DESCRIPTION

This module implements a multi-process and multi-thread load generator for HTTP. It uses Coro threads. So, in reality it does not use threads but event-based IO.

Features

• limited support for SSL connections

• keep-alive connections

• configurable delay before and after each request

• run a list of URLs many times

• compute next URL based on the current request

• DNS cache can be preinitialized

• slow ramp up

• request bodies

• custom request headers

Overview

Note, this POD is best view via Apache2::PodBrowser.

Parallelism

The load generator follows a 2-level supervisor-worker pattern. The central function, loadgen, creates a certain number of child processes. Each child process then creates in a slow ramp up phase worker threads up to a configurable total upper thread limit.

The thread limit is configured independent on the number of worker processes. You configure a number of processes that is about 1.5-5 times the number of available CPUs. The number of threads can then be say 50 or 500 or even 5996 or so. Processes and threads are numbered starting from 0.

So, assuming there are 3 processes and 10 threads configured the following table shows how the threads are spread among the processes:

Process | Threads
--------+------------
      0 | 0 3 6 9
      1 | 1 4 7
      2 | 2 5 8

Process 0 will run 4 threads, the other 2 processes 3 threads each. The number of threads per process can be calculated as:

$TotalThreadCount / $NProc + ($ProcNr < $TotalThreadCount % $NProc)

where $NProc is the number of processes used, $ProcNr the number of the current process and $TotalThreadCount the system-wide thread number. $ProcNr ranges from 0 to $NProc - 1.

At the beginning ot the ramp-up phase each process starts up a certain number of threads (maybe 0) to reach the configured start-up thread number. The configured ramp-up duration then determines in which intervals new threads are added. So assuming the threads run long enough you start up with a certain level of parallelism which increases linearly over a certain time interval up to the configured maximum.

The Scoreboard

The multi-process model of HTTP::LoadGen means that each process knows only about its own threads. Sometimes you may want to log for example the overall number of active requests when a new request is started. Or you may want to increment a shared variable for each request to see the progress of an active load run.

HTTP::LoadGen::ScoreBoard or IPC::ScoreBoard may be used to achieve that.

The Logger

HTTP::LoadGen doesn't have logging built-in. Instead HTTP::LoadGen::Logger is provided.

Random numbers and repeatable results

loadgen needs for certain operations random numbers. If you need repeatable results that is you want to repeat the same test with the same delays between requests later then you need the same sequence of random numbers for each thread. But the random number generator built-in to Perl is process-wide.

HTTP::LoadGen provides an interface to set an RNG per thread. CPAN modules like Math::Random::MT use an object oriented approach. So, it may be a good idea to create such an object for each thread and register it with HTTP::LoadGen. A ThreadInit handler is a good place to do that.

If the import function of HTTP::LoadGen is called with the -rand parameter (use HTTP::LoadGen qw/-rand/) the Perl built-in rand operator is overwritten (by means of *CORE::GLOBAL::rand) to use the thread-specific RNG. Though, occurences of rand in the code that have been compiled before HTTP::LoadGen is loaded continue to use the built-in operator.

Phases

There are several phases in the lifetime of a load run, a process, a thread or a request that can be hooked. A hook is a code reference.

Process related phases

ParentInit and ParentExit

these 2 hooks run in the parent process. The loadgen function checks the configuration and then calls ParentInit. ParentExit is called just before loadgen returns.

ParentInit can start Coro threads. They will run while the process is waiting for the worker children to finish.

One thing to consider to do in a ParentInit hook is the creation of a HTTP::LoadGen::ScoreBoard.

ProcInit and ProcExit

these 2 hooks are called in each worker process. When a worker process is started ProcInit is called. But before the actual load generation is started the process waits for a signal from the parent process that is sent when every worker process has finished its ProcInit phase. So, even if the ProcInit phase takes a bit longer it does not influence the load generation other than it is started a bit later.

ProcInit can start Coro threads. They will run while the process is waiting for the signal from the parent process to start load generation and of course after that until they finish.

One thing that should probably be done in a ProcInit handler is reseeding of the random number generator. If you need repeatable results then you need a random number generator per thread. The built-in RNG is no help then. However, there are several object oriented RNGs on CPAN. Use the rng function to set a thread-specific RNG and rnd instead of the built-in rand to call it.

Another one is the creation of a logger, see HTTP::LoadGen::Logger.

ProcExit is called after the load generation is over just before the worker process exists.

If a ProcExit hook is installed its return value determines the exit code of the worker process.

Closing the logger would be good here.

Thread related phases

ThreadInit and ThreadExit

these 2 hooks wrap the load generation phase of each thread. If a thread needs private data the ThreadInit handler can create and return it. It is then passed to all other hooks called during the lifetime of the thread.

Things to consider to do in a ThreadInit handler would include

1. registration of a thread-specific random number generator

2. registration of the thread with the scoreboard

InitURLs

during the load generation phase each thread fetches a list of URLs several times. The actual list is not given as an array or similar but as an interator generator, that is a function that returns a function that returns an URL to be fetched. The InitURLs iterator generator is called each time a thread starts another round of fetching URLs. The iterator itself is then called to get the next URL to be fetched. If it returns undef or the empty list the current round is over. Then if the configured number of rounds is reached the thread ends or the next round is started (and the InitURLs handler is called again).

In most cases this complex URL handling is not necessary. Instead one simply needs to check off all items of a predefined list. For these situations a few predefined iterator generators exist.

ReqStart and ReqDone

these 2 hooks wrap each HTTP request.

Here the request would be accounted with the scoreboard. In ReqDone logging would occur.

HTTP::LoadGen::loadgen \%data

loadgen is the central function of this module. It starts up child processes, creates threads, generates the load and waits for that all to finish. It returns when all is done.

The %data hash passed by reference configures loadgen and describes what to do. loadgen copies the hash so that the original hash is not changed but the copying is not recursive. If a hash value is an array and one of the hooks changes it that change will be reflected in the original %data hash. However, if you add new hash elements in a hook function they won't show up in %data after loadgen returns.

Request descriptor and return element

A number of elements of the %data hash are hook functions. Some of them are passed parametes $rq and/or $rc. Both are lists. HTTP::LoadGen::Run exports constants to access the list elements. The structure of the request descriptor $rq is explained under URLList below.

For the lack of a better place the $rc element is described here.

RC_STATUS (0)

the HTTP status code. If the request failed because the connection couldn't be established a code 599 is set here. RC_STATUSLINE describes the problem in more detail in that case.

RC_STATUSLINE (1)

the HTTP status message. If the server responds with the following first line for example:

HTTP/1.1 501 Method Not Implemented

RC_STATUS is 501 while RC_STATUSLINE is Method Not Implemented.

RC_HTTPVERSION (2)

the server HTTP protocol version. Normally 1.1 or 1.0.

RC_STARTTIME (3)

when the request has been started, fractional number.

RC_CONNTIME (4)

when the connection has been established, fractional number.

RC_FIRSTTIME (5)

when the first line of output has been received, fractional number.

RC_HEADERTIME (6)

when the response HTTP header has been completely received, fractional number.

RC_BODYTIME (7)

when the response body has been completely received, fractional number.

RC_HEADERS (8)

a hash containing the response HTTP headers. The values of this hash are arrays since HTTP header fields can be given multiple times. Keys (header names) are converted to lower case.

Example:

{
 'content-type' => ['text/html; charset=iso-8859-1'],
 'connection' => ['close'],
 'date' => ['Sun, 04 Jul 2010 18:21:12 GMT'],
 'content-length' => ['217'],
 'allow' => ['GET,HEAD,POST,OPTIONS,TRACE'],
 'server' => ['Apache'],
}

RC_BODY (9)

the response body

RC_DNSCACHED (10)

boolean: has the DNS cache lookup resulted in a hit (1) or miss (0)?

RC_CONNCACHED (11)

boolean: has the has a kept-alive connection been used?

The %data hash

So, what can be specified in %data? Note, all keys here are case sensitive.

NWorker (optional)

specifies the number of worker processes to be used. Default is 1.

RampUpStart (optional)

the number of threads to started up immediately (after the ProcInit phase is over). Default is 1 thread per worker process, that is NWorker.

RampUpMax (optional)

the number of threads that have to be started up after the ramp-up phase is over. That means all processes together will start this number of threads. If a thread finishes before the ramp-up phase is over this maximum level of parallelism will never be reached.

Default is the same as RampUpStart.

RampUpDuration (optional)

the duration of the ramp-up phase in seconds (may be fraction).

Default is 300 (5 minutes).

ParentInit (optional)

the ParentInit handler called as

$data->{ParentInit}->();

One thing to do here is to create a scoreboard for interprocess communication, see HTTP::LoadGen::ScoreBoard or IPC::ScoreBoard.

Example:

ParentInit=>sub {
  # no parameters

  # create scoreboard
  # options() returns the config hash itself. The NWorker parameter
  # is known. SbSlotsz and SbExtra are new. This is to demonstrate
  # that the hook routines can access the configuration and evaluate
  # and even add custom parameters.
  HTTP::LoadGen::ScoreBoard::init_once
      @{HTTP::LoadGen::options()}{qw/NWorker SbSlotsz SbExtra/};
}

ParentExit (optional)

the ParentExit handler called as

$data->{ParentExit}->();

If a scoreboard is used remember to disconnect.

Example:

ParentExit=>sub {
  # no parameters
  undef HTTP::LoadGen::ScoreBoard::scoreboard;
}

ProcInit (optional)

the ProcInit handler called as

$data->{ProcInit}->($procnr);

$procnr is the 0 based number of the process. It ranges up to NWorker - 1.

If you plan to use the built-in random number generator this hook is a good place to reseed it.

Another good thing to do here is to acquire a logger. HTTP::LoadGen::Logger may help here.

If a scoreboard is used save $procnr as slot number.

Example:

ProcInit=>sub {
  my ($procnr)=@_;

  # set my slot number
  HTTP::LoadGen::ScoreBoard::slot=$procnr;

  # acquire a logger
  $logger=HTTP::LoadGen::Logger::get;
}

ProcExit (optional)

the ProcExit handler called as

$rc=$data->{ProcExit}->($procnr);

The return value of this hook determines the exit code of the process. If omitted the exit code is 0.

The thing to do here is perhaps to close the logger.

Example:

ProcExit=>sub {
  my ($procnr)=@_;
  $logger->();		# close the logger
}

ThreadInit (optional)

the ThreadInit handler called as

$userdata=$data->{ThreadInit}->();

The return value of this hook is saved as thread-specific user data.

This hook is a good place to initialize a thread specific random number generator if you need repeatable results.

Example:

ThreadInit => sub {
  # no parameters

  # thread accounting with the scoreboard
  HTTP::LoadGen::ScoreBoard::thread_start;

  # set a thread specific RNG
  HTTP::LoadGen::rng=Math::Random::MT->new(@seed);

  return [];		  # initializes thread specific user data
}

ThreadExit (optional)

the ThreadExit handler called as

$data->{ThreadExit}->();

Remember to notify the scoreboard.

Example:

ThreadExit=>sub {
  # no parameters
  HTTP::LoadGen::ScoreBoard::thread_done;
}

ReqStart (optional)

the ReqStart handler called as

$data->{ReqStart}->($rq);

$rq is an array specifying the current request. It is generated by the URL iterator. The ReqStart handler is allowed modify the array.

If a scoreboard is used check in the request. One can also save some current state from the scoreboard to the thread-specific storage to log it later.

Example:

 ReqStart=>sub {
   my ($rq)=@_;
   HTTP::LoadGen::ScoreBoard::req_start;
   @{HTTP::LoadGen::userdata()}=(HTTP::LoadGen::ScoreBoard::thread_count,
				 HTTP::LoadGen::ScoreBoard::req_started,
				 HTTP::LoadGen::ScoreBoard::req_success,
				 HTTP::LoadGen::ScoreBoard::req_failed);
 }

ReqDone (optional)

the ReqDone handler called as

$data->{ReqDone}->($rc, $rq);

$rc is the request result. See run_url in HTTP::LoadGen::Run. The ReqDone handler may modify this array. But it's not recommended to do that.

$rq is an array specifying the current request.

Here you do request accounting and of course logging.

Example:

 ReqDone=>sub {
   my ($rc, $rq)=@_;

   HTTP::LoadGen::ScoreBoard::req_done
       scalar($rc->[RC_STATUS]=~/^[23]/), $rc->[RC_HEADERS], $rc->[RC_BODY];

   $logger->(HTTP::LoadGen::threadnr,
	     @{$rc}[RC_DNSCACHED, RC_CONNCACHED],
	     @{HTTP::LoadGen::userdata()},
	     HTTP::LoadGen::ScoreBoard::req_success,
	     HTTP::LoadGen::ScoreBoard::req_failed,
	     HTTP::LoadGen::ScoreBoard::header_count,
	     HTTP::LoadGen::ScoreBoard::header_bytes,
	     HTTP::LoadGen::ScoreBoard::body_bytes,
	     $rc->[RC_STARTTIME],
	     $rc->[RC_CONNTIME]-$rc->[RC_STARTTIME],
	     $rc->[RC_FIRSTTIME]-$rc->[RC_STARTTIME],
	     $rc->[RC_HEADERTIME]-$rc->[RC_STARTTIME],
	     $rc->[RC_BODYTIME]-$rc->[RC_STARTTIME],
	     $rc->[RC_STATUS], $rc->[RC_STATUSLINE],
	     length($rc->[RC_BODY]),
	     sprintf('%s(%s://%s:%s%s)',
		     @{$rq}[RQ_METHOD, RQ_SCHEME, RQ_HOST, RQ_PORT, RQ_URI]));
 }

times (optional)

the number of times the URL iterator is charged. That many times the URL list is fetched.

If omitted or <=0 the test runs forever.

dnscache (optional)

loadgen caches DNS query results. One can prevent DNS queries completely in 2 ways. One of them is to provide a hash here that maps names to IP addresses. The other is to have the URL iterator generate IP addresses instead of host names and optionally Host request header fields.

Another use of this item is to cheat host name resolution. One can for example test a newly installed or development server while the real server continues to work unaffected.

Example:

 dnscache=>{
	    'foertsch.name'=>'127.0.0.1',
	   },

InitURLs (either InitURLs or URLList or both must be present)

InitURLs initializes the URL iterator. It may be a string describing one of the predefined iterators or a CODE reference.

In the latter case it is called without parameters as

$it=$data->{InitURLs}->();

It is expected to return a function that when called as

$new_rq=$it->($rc, $rq);

returns the next request item or undef when it runs out of items. The parameters $rc and $rq describe the previous request ($rq) and its result ($rc).

For a description of the $rq and $new_rq format see URLList below.

Example:

 InitURLs=>sub {
   my $url=[qw!GET http foertsch.name 80 /-redir!,
	    {
	     keepalive=>KEEPALIVE,
	     headers=>[
		       'X-auth'=>1, # necessary to trigger 401 for that URL
		      ],            # it also shows a custom request header
	    }];
   return sub {
     my ($rc, $rq)=@_;
     if( $rc->[RC_STATUS]==401 ) {
       # redo with Authorization header
       push @{$rq->[RQ_PARAM]->{headers}}, Authorization=>'Basic YmxhOmJsdWI=';
       return $rq;
     }
     my $new_rq=$url;
     undef $url;		# next time return undef (out-of-requests)
     return $new_rq;
   };
 }

The iterator generator initializes the variable $url and then returns a closure. Hence, $url is a static variable with respect to the returned iterator.

The iterator itself checks the HTTP code of the previous request. In case of a 401 (Authorization Required) it adds an Authorization header to the request header list and retries the operation.

If the previous operation has ended with an other HTTP code it copies $url to an auxiliary variable, undefines it and returns the auxiliary variable. Thus, only the first time the iterator is called it returns $url. After that it is always undef which signals Out-of-Requests.

If InitURLs is a string it is the name of a predefined iterator generator.

Example:

InitURLs=>'follow'

There are currently 4 such generators. All of them expect an URLList (see below) to be provided.

default

simply walks the URLList from start to end.

This one is also used if InitURLs is omitted.

random_start

similar to default but starts at a random offset in URLList. At the end of the list it continues at the beginning until all URLList elements are done once.

follow

similar to default but if a request results in a 3xx HTTP code and a Location header is provided by the server it tries to follow it recursively.

If the request starting a series of redirections contains a postdelay statement (see below) the delay is postponed until after the last request of the series. Subsequent requests are issues without delay.

Subsequent requests inherit the User-Agent and Referer HTTP headers of the originating request, see follow_3XX() below.

This iterator is a bit special in that can turn other iterators into following ones. Normally an iterator generator is called without parameters. This one can take one parameter that in turn may be an iterator. It returns then a following iterator based on the passed one.

Infact, the built-in random_start_follow iterator is implemented for example as

register_iterator random_start_follow=>sub {
   @_=get_iterator('random_start')->();
   goto &{get_iterator 'follow'};
};

To turn your own iterator into a following you could write:

InitURLs=>sub {
  return get_iterator->('follow')->($my_own_iterator);
}

where $my_own_iterator is an iterator function.

random_start_follow

a combination of the 2 above.

You can register your own named iterators by calling register_iterator below.

URLList (either InitURLs or URLList or both must be present)

See also InitURLs above.

An URLList is an array of arrays. Each of these sub-arrays describes one request. If consists of 6 elements:

[$method, $scheme, $host, $port, $uri, $param]

$method is the HTTP request method, e.g. GET, POST, ...

$scheme is either http or https.

$host is the hostname or IP address of the server, e.g. foertsch.name or 109.73.51.50.

$port is the server port to connect. Usually port 80 is used for http and port 443 for https.

$uri is the request URI normally starting with a slash (/), e.g. /impressum.html.

$param is a hash with further options.

To access the elements of a request description HTTP::LoadGen::Run exports a few constants. They may be used to increase readability.

RQ_METHOD == 0
RQ_SCHEME == 1
RQ_HOST   == 2
RQ_PORT   == 3
RQ_URI    == 4
RQ_PARAM  == 5

Example:

 URLList=>[
           [qw!GET http 109.73.51.50 80 /-redir!,
	    {
	     keepalive=>KEEPALIVE,
	     headers=>[
		       Authorization=>'Basic YmxhOmJsdWI=',
                       Host=>'foertsch.name',
		      ],
	    }],
           [qw!HUGO https www.kabatinte.net 443 /!,
	    {
	     keepalive=>KEEPALIVE,
             predelay=>0.5,
             prejitter=>1,
             postdelay=>3,
             postjitter=>1.5,
             body=>'blablub',
	    }]
          ]

This URLList contains 2 requests, one for a server with the IP address 109.73.51.50 and one for the host www.kabatinte.net.

The first one will send the following HTTP request to the server (IP 109.73.51.50, port 80):

GET /-redir HTTP/1.1
Authorization: Basic YmxhOmJsdWI=
Host: foertsch.name

If you need more header fields, User-Agent for example, add them to the headers array of the options hash.

The second request is converted into the following HTTP message sent over SSL to 84.38.75.176:443 assuming that www.kabatinte.net resolves to 84.38.75.176:

HUGO / HTTP/1.1
Host: www.kabatinte.net
Content-Length: 7

blablub

Although no Host header is specified in the request element one is sent. If the request element does not contain a Host header one is added automatically based on $host and $port.

You may also notice the Content-Length header. It is sent because a request body is specified (the body item in $param).

So, what can be specified in the $param part?

keepalive

HTTP::LoadGen::Run exports 3 constants to be used as values. KEEPALIVE_USE permits to use a previously kept alive connection. KEEPALIVE_STORE allows to keep the connection alive after the request. KEEPALIVE combines both of the above.

If you hate readability you can also use the numerical values:

KEEPALIVE_USE==1
KEEPALIVE_STORE==2
KEEPALIVE==3

predelay and prejitter

These statements define a period to wait before sending the request. The wait is done after the request description has been pulled off the iterator but before the ReqStart handler is run.

Both numbers can be fractions. Read them as

predelay ± prejitter

The actual waiting time is calculated as

interval = predelay - prejitter + rand( 2 * prejitter )

If prejitter >= predelay interval can become negative. In this cases you won't jump back in time but simply not wait.

To achieve repeatable results a thread-specific random number generator must be used. See the rng function below.

postdelay and postjitter

The same as predelay but waiting occurs after the request is done or more precisely after the ReqDone handler returns.

headers

an array (not a hash!) of header fields to be appended to the HTTP request.

body

a request body

conn_timeout

here you can specify the return value of the prepare-callback function passed to AnyEvent::Socket::tcp_connect when establishing a connection.

See AnyEvent::Socket for more information.

timeout

the timeout parameter used when a connection is converted into a AnyEvent::Handle object.

See AnyEvent::Handle for more information.

tls_ctx

the tls_ctx parameter used when a connection is converted into a AnyEvent::Handle object.

See AnyEvent::Handle for more information.

By now AnyEvent::Handle supports SSL features like client certificates and server certificate verification. However, some things are still missing like SSL session caching. How about server initiates renegotiations I am not sure.

Note, conn_timeout, timeout and tls_ctx are not very well tested by now.

Useful functions to be used in hooks

HTTP::LoadGen::threadnr

returns the number of the thread currently running.

HTTP::LoadGen::userdata

returns the thread-specific user data. Normally this is assigned to by returning something useful from a ThreadInit handler. But it's a lvalue-function. Hence the following will work too:

# assign new thread-specific data
HTTP::LoadGen::userdata={something=>'useful'};

HTTP::LoadGen::options

returns the copy of the configuration hash used by loadgen().

HTTP::LoadGen::rng

returns and sets the thread-specific random number generator. It sets the RNG used by HTTP::LoadGen::rnd.

HTTP::LoadGen::rnd $upper_limit

use the thread-specific random number generator or if none set the built-in one.

Returns a pseudo-random number.

HTTP::LoadGen::delay $prefix, $param

this function implements the predelay and postdelay operations.

$prefix is a prefix, e.g. pre or post.

$param is a RQ_PARAM hash of a request descriptor containing keys $prefix.'delay' and $prefix.'jitter'.

HTTP::LoadGen::done

if a thread needs a preliminary exit call

HTTP::LoadGen::done=1

in a ReqStart or ReqDone handler. The current request will be performed except for postdelay. Then the thread finishes. This can be used to stop the run when a certain load level has been reached.

$new_rq=HTTP::LoadGen::follow_3XX $rc, $rq

This function implements the following part of the built-in follow iterator.

It is called with the result and the request descriptor of the previous request and returns a new request descriptor if the result is a HTTP redirect.

Otherwise an empty list is returned.

The new request preserves the User-Agent and Referer request header fields.

Other auxiliary functions

HTTP::LoadGen::register_iterator $name, $code_ref

registers a known iterator. This can be used by other modules, e.g.

package My::Iterator;
use HTTP::LoadGen ':all';
BEGIN {
  register_iterator 'my_iterator'=>sub {
    ...
  };
}

$code_ref=HTTP::LoadGen::get_iterator $name

returns an iterator by name.

$handle=HTTP::LoadGen::create_proc $nproc, $init_hnd, $hnd, $exit_hnd

create $nproc child processes and have them finish the ProcInit phase $init_hnd.

$init_hnd and $hnd are passed just one parameter, the 0-based process number. $exit_hnd get that plus the scalar return value of $hnd. The return value of $exit_hnd determines the exit code of the child process.

$init_hnd and $exit_hnd may be undef. $hnd may not.

Returns an opaque handle that can be passed to start_proc.

$status=HTTP::LoadGen::start_proc $handle

When create_proc returns all children have finished their $init_hnd and wait for a signal to continue with $hnd. start_proc sends that signal and waits for all children to finish.

It returns a hash that maps operating system process IDs to their exit code, killing signal and a coredump flag.

Example:

{
 '7273' => [7, 0, 0],     # PID 7273 exits normally with code 7
 '7275' => [0, 11, 1],    # PID 7275 has been killed by signal 11
                          #   + core has been dumped
 '7274' => [8, 0, 0],     # PID 7274 exits normally with code 8
}

$semaphore=HTTP::LoadGen::ramp_up $procnr, $nproc, $start, $max, $duration, $handler

implements the ramp-up phase.

returns a semaphore that can be used to wait for the created threads to finish. It waits only for the threads running in the current process:

$semaphore->down;      # wait for my threads to finish

ramp_up may finish almost immediately but may also take some time while the load generation is already running. It depends on the $duration parameter. Don't expect it to return before the load generation starts.

Internal functions

HTTP::LoadGen::conncache

HTTP::LoadGen::tlscache

DEBUGGING

Sometimes its useful to see what requests are made. If the environment variable HTTP__LoadGen__Run__dbg is set when HTTP::LoadGen::Run is compiled a source filter is used to compile in debugging output to STDERR.

EXPORT

The following Exporter tags are defined:

common

exports loadgen, threadnr, done, userdata, options, rng, rnd and delay

const

exports all symbols that HTTP::LoadGen::Run exports by default.

all

all of the above.

Additionally it pulls in HTTP::LoadGen::ScoreBoard and exports all that is exported by it.

Also, HTTP::LoadGen::ScoreBoard is loaded. Then function named get_logger is created as an alias for HTTP::LoadGen::Logger::get and exported.

EXAMPLE CONFIGURATION

#!/usr/bin/loadgen
# -*-perl-*-

use strict;
use Math::Random::MT;
use Coro;
use Coro::Timer ();
no warnings 'ambiguous';

# possible hook parameters:
# $procnr   -- the current process number 0 .. NWorker-1
# $el       -- an URL element to fetch (ARRAY)
#              use RQ_* constants from HTTP::LoadGen::Run to access
# $rc       -- an result element (ARRAY)
#              use RC_* constants from HTTP::LoadGen::Run to access

my $logger;
+{
  NWorker=>3,	                # use 3 processes
  RampUpStart=>2,              # start 2 threads immediately
  RampUpMax=>13,      	        # then add 11 threads over 5 seconds
  RampUpDuration=>5,           # that makes 2.2 new threads per second

  ParentInit=>sub {
    # no parameters

    # create scoreboard
    sbinit undef, options->{NWorker};
  },
  ParentExit=>sub {
    # no parameters
    undef scoreboard;
  },

  ProcInit=>sub {
    my ($procnr)=@_;

    # set my slot number
    slot=$procnr;

    # acquire a logger
    my $fmt='%-2d %d %d %2d %2d %3d %3d %.3f %.3f %.3f %.3f %.3f %s %d '.
            "%s(%s://%s:%s%s) %s\n";
    $logger=get_logger undef, sub {sprintf $fmt, @_};
  },
  ProcExit=>sub {
    my ($procnr)=@_;
    $logger->();               # close the logger
  },

  ThreadInit=>sub {
    # no parameters

    # thread accounting
    thread_start;

    # set a thread specific RNG
    rng=Math::Random::MT->new(threadnr);

    return [];                 # initializes thread specific user data
  },
  ThreadExit=>sub {
    # no parameters
    thread_done;
  },

  ReqStart=>sub {
    my ($el)=@_;

    # request accounting
    req_start;

    # started - succeeded - failed = currently pending number of requests
    @{userdata()}=(thread_count, req_started-req_success-req_failed);
  },

  ReqDone=>sub {
    my ($rc, $el)=@_;

    # request accounting: HTTP status 2xx and 3xx are successful
    #                     other requests are counted as failures.
    req_done +($rc->[RC_STATUS]=~/^[23]/), $rc->[RC_HEADERS], $rc->[RC_BODY];

    $logger->(threadnr,
              @{$rc}[RC_DNSCACHED, RC_CONNCACHED],
              @{userdata()},
              req_success,
              req_failed,
              $rc->[RC_STARTTIME],
              $rc->[RC_CONNTIME]-$rc->[RC_STARTTIME],
              $rc->[RC_FIRSTTIME]-$rc->[RC_STARTTIME],
              $rc->[RC_HEADERTIME]-$rc->[RC_STARTTIME],
              $rc->[RC_BODYTIME]-$rc->[RC_STARTTIME],
              $rc->[RC_STATUS],
              length($rc->[RC_BODY]),
              @{$el}[RQ_METHOD, RQ_SCHEME, RQ_HOST, RQ_PORT, RQ_URI],
              $rc->[RC_STATUSLINE]);
  },

  dnscache=>{
             localhost=>'127.0.0.1',
             'kabatinte.net'=>'84.38.75.176',
             'www.kabatinte.net'=>'84.38.75.176',
             'foertsch.name'=>'109.73.51.50',
            },

  times=>3,			# run the URL list 3 times

  InitURLs=>'random_start',

  URLList=>do {
    my $o={
           keepalive=>KEEPALIVE,
           qw!predelay 0.05 prejitter 0.1 postdelay 0.5 postjitter 1!,
          };
    [[qw!GET http foertsch.name 80 /-redir!, $o],
     [qw!HUGO https www.kabatinte.net 443 /!, $o]
    ];
  },
 }

SEE ALSO

• HTTP::LoadGen::Run

• HTTP::LoadGen::ScoreBoard

• HTTP::LoadGen::Logger

• loadgen

AUTHOR

Torsten Förtsch, <torsten.foertsch@gmx.net>

COPYRIGHT AND LICENSE

Copyright (C) 2010 by Torsten Förtsch

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.10.0 or, at your option, any later version of Perl 5 you may have available.

cpanm HTTP::LoadGen

perl -MCPAN -e shell
install HTTP::LoadGen