/ 
Test-LectroTest-0.2005
River stage three • 5 direct dependents • 287 total dependents
/ Test::LectroTest::TestRunner

NAME

Test::LectroTest::TestRunner - Configurable Test::Harness-compatible engine for running LectroTest property checks

SYNOPSIS

use Test::LectroTest::TestRunner;

my @args = trials => 1_000, retries => 20_000;
my $runner = Test::LectroTest::TestRunner->new( @args );

# test a single property and print details upon failure
my $result = $runner->run( $a_single_lectrotest_property );
print $result->details unless $result->success;

# test a suite of properties, w/ Test::Harness output
my $all_successful = $runner->run_suite( @properties );
print "Splendid!" if $all_successful;

DESCRIPTION

STOP! If you just want to write and run simple tests, see Test::LectroTest.

This module provides Test::LectroTest::TestRunner, a class of objects that test properties by running repeated random trials. You create a TestRunner, configure it, and then call its run or run_suite methods to test properties individually or in groups.

METHODS

The following methods are available.

new
my $runner = new Test::LectroTest::TestRunner(
  trials  => 1_000,
  retries => 20_000,
  scalefn => sub { $_[0] / 2 + 1 },
  verbose => 1
);

Creates a new Test::LectroTest::TestRunner and configures it with the given named parameters, if any. Typically, you need only provide the trials parameter because the other values are reasonable for almost all situations. Here is what each parameter means:

trials

The number of trials to run against each property checked. The default is 1_000.

retries

The number of times to allow a property to retry trials (via $tcon-retry>) during the entire property check before aborting the check. This is used primary to prevent infinite looping, should the property retry every attempt.

scalefn

A subroutine that scales the sizing guidance given by the TestRunner to the input generators.

The TestRunner starts with an initial guidance of 1 at the beginning of a property check. For each trial (or retry) of the property, the guidance value is incremented. This causes successive trials to be tried using successively larger inputs. The scalefn subroutine gets to adjust this guidance on the way to the input generators. Typically, you would change the scalefn subroutine if you wanted to change the rate and which inputs grow during the course of the trials.

verbose

If true (the default) the TestRunner will use verbose output that includes things like label frequencies and counterexamples. Otherwise, only one-line summaries will be output. Unless you have a good reason to do otherwise, leave this parameter alone because verbose output is almost always what you want.

You can also set and get the values of the configuration properties using accessors of the same name.

run
$results = $runner->run( $a_property );
print $results->summary, "\n";
if ($results->success) {
    # celebrate!
}

Checks whether the given property holds by running repeated random trials. The result is a Test::LectroTest::TestRunner::results object, which you can query for fined-grained information about the outcome of the check.

The run method takes an optional second argument which gives the test number. If it is not provided (usually the case), the next number available from the test runner's internal counter is used.

$results = $runner->run( $third_property, 3 );
run_suite
my $all_success = $runner->run_suite( @properties );
if ($all_success) {
    # celebrate most jubilantly!
}

Checks a suite of properties, sending the results of each property checked to STDOUT in a form that is compatible with Test::Harness. For example:

1..5
ok 1 - Property->new disallows use of 'tcon' in bindings
ok 2 - magic Property syntax disallows use of 'tcon' in bindings
ok 3 - exceptions are caught and reported as failures
ok 4 - pre-flight check catches new w/ no args
ok 5 - pre-flight check catches unbalanced arguments list

By default, labeling statistics and counterexamples (if any) are included in the output. You may turn them off by passing verbose=>0 after all of the properties in the argument list.

Test::LectroTest::TestRunner::results

my $results = $runner->run( $a_property );
print "Property name: ", $results->name, ": ";
print $results->success ? "Winner!" : "Loser!";

This is the object that you get back from run. It contains all of the information available about the outcome of the property check and provides the following methods:

success

Boolean value: True if the property checked out successfully; false otherwise.

summary

Returns a one line summary of the property-check outcome. It does not end with a newline. Example:

ok 1 - Property->new disallows use of 'tcon' in bindings
details

Returns all relevant information about the property-check outcome as a series of lines. The last line is terminated with a newline. The details are identical to the summary (except for the terminating newline) unless label frequencies are present or a counterexample is present. Example:

1..1
not ok 1 - 'my_sqrt meets defn of sqrt' falsified in 1 attempts
# Counterexample:
# $x = '0.546384454460178';
name

Returns the name of the property to which the results pertain.

number

The number assigned to the property that was checked.

counterexample

Returns the counterexample that "broke" the code being tested, if there is one. Otherwise, returns an empty string.

labels

Label counts. If any labels were applied, this value will be a reference to a hash mapping each combination of labels to the count of trials that had that particular combination. Otherwise, it will be undefined.

Note that each trial is counted only once -- for the most-specific combination of labels that are applied to it. For example, consider the following labeling logic:

Property {
  ##[ x <- Int ]##
  $tcon->label("negative") if $x < 0;
  $tcon->label("odd")      if $x % 2;
  1;
}, name => "negative/odd";

For a particular trial, if x was 2 (positive and even), the trial would receive no labels. If x was 3 (positive and odd), the trial would be labeled "odd". If x was -2 (negative and even), the trial would be labeled "negative". If x was -3 (negative and odd), the trial would be labeled "negative & odd".

label_frequencies

Returns a string containing a line-by-line accounting of labels applied during the series of trials:

print $results->label_frequencies;

The corresponding output looks like this:

25% negative
25% negative & odd
25% odd

If no labels were applied, an empty string is returned.

exception

Returns the text of the exception that caused the test series to be aborted, if there is one. Otherwise, returns an empty string.

attempts

Returns the count of trials performed.

incomplete

In the event that the series of trials was halted before it was completed (such as when the retry count was exhausted), this method will return the reason. Otherwise, it returns an empty string.

Note that a series of trials is complete if a counterexample is found.

Test::LectroTest::TestRunner::testcontroller

During a live property-check trial, the variable $tcon is available to your Properties. It lets you label the current trial or request that it be re-tried with new inputs.

The following methods are available.

retry
Property {
  ##[ x <- Int ]##
  return $tcon->retry if $x == 0; 
}, ... ;

Stops the current trial and tells the test runner to re-try it with new inputs. Typically used to reject a particular case of inputs that doesn't make for a good or valid test.

label
Property {
  ##[ x <- Int ]##
  $tcon->label("negative") if $x < 0; 
  $tcon->label("odd")      if $x % 2; 
}, ... ;

Applies a label to the current trial. At the end of the trial, all of the labels are gathered together, and the trial is dropped into a bucket bearing the combined label. See the discussion of "labels" for more.

trivial
Property {
  ##[ x <- Int ]##
  $tcon->trivial if $x == 0; 
}, ... ;

Applies the label "trivial" to the current trial. It is identical to calling label with "trivial" as the argument.

LECTROTEST HOME

The LectroTest home is http://community.moertel.com/LectroTest. There you will find more documentation, presentations, a wiki, and other helpful LectroTest-related resources. It's also the best place to ask questions.

AUTHOR

Tom Moertel (tom@moertel.com)

INSPIRATION

The LectroTest project was inspired by Haskell's fabulous QuickCheck module by Koen Claessen and John Hughes: http://www.cs.chalmers.se/~rjmh/QuickCheck/.

COPYRIGHT and LICENSE

Copyright (c) 2004 by Thomas G Moertel. All rights reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.