/ 
Test2-Aggregate-0.08
River stage zero No dependents
/ Test2::Aggregate

NAME

Test2::Aggregate - Aggregate tests

SYNOPSIS

use Test2::Aggregate;

Test2::Aggregate::run_tests(
    dirs => \@test_dirs
);

done_testing();

VERSION

Version 0.08

DESCRIPTION

Aggregates all tests specified in dirs (which can even be individual tests), to avoid forking, reloading etc that can help with performance or profiling. Test files are expected to end in .t and are run as subtests of a single aggregate test.

A bit similar but simpler in concept and execution than Test::Aggregate, which makes it more likely to work with your test suite and also with more modern Test2 bundles. It does not try to package each test which may be good or bad (e.g. redefines), depending on your requirements.

Generally, the way to use this module is to try to aggregate sets of quick tests (e.g. unit tests). Try to iterativelly add tests to the aggregator dropping those that do not work.

METHODS

run_tests

my $stats = Test2::Aggregate::run_tests(
    dirs          => \@dirs,              # optional if lists defined
    lists         => \@lists,             # optional if dirs defined
    root          => '/testroot/',        # optional
    load_modules  => \@modules,           # optional
    shuffle       => 0,                   # optional
    reverse       => 0,                   # optional
    repeat        => 1,                   # optional, requires Test2::Plugin::BailOnFail for < 0
    slow          => 0,                   # optional
    override      => \%override,          # optional, requires Sub::Override
    stats_output  => $stats_output_path,  # optional, requires Time::HiRes
    extend_stats  => 0,                   # optional
    test_warnings => 0                    # optional
);

Runs the aggregate tests. Returns a hashref with stats like this:

$stats = {
  'test.t' => {
    'test_no'   => 1,                 # numbering starts at 1
    'pass_perc' => 100,               # for single runs pass/fail is 100/0
    'timestamp' => '20190705T145043', # start of test
    'time'      => '0.1732'           # only with stats_output
  }
};

The parameters to pass:

  • dirs (either this or lists is required)

    An arrayref containing directories which will be searched recursively, or even individual tests. The directories (unless shuffle or reverse are true) will be processed and tests run in order specified.

  • lists (either this or dirs is required)

    Arrayref of flat files from which each line will be pushed to dirs (so they have a lower precedence - note root still applies).

  • root (optional)

    If defined, must be a valid root directory that will prefix all dirs and lists items. You may want to set it to './' if you want dirs relative to the current directory and the dot is not in your @INC.

  • load_modules (optional)

    Arrayref with modules to be loaded (with eval "use ...") at the start of the test. Useful for testing modules with special namespace requirements.

  • override (optional)

    Pass Sub::Override key/values as a hashref.

  • repeat (optional)

    Number of times to repeat the test(s) (default is 1 for a single run). If repeat is negative, the tests will repeat until they fail (or produce a warning if test_warnings is also set).

  • shuffle (optional)

    Random order of tests if set to true.

  • reverse (optional)

    Reverse order of tests if set to true.

  • slow (optional)

    When true, tests will be skipped if the environment variable SKIP_SLOW is set.

  • test_warnings (optional)

    Tests for warnings over all the tests if set to true. It will print an array of warnings, however if you want to see the warnings the moment they are generated (for debugging etc), then leave it disabled.

  • stats_output_path (optional)

    stats_output_path when defined specifies a path where a file with running time per test (average if multiple iterations are specified), starting with the slowest test and passing percentage gets written. On negative repeat the stats of each successful run will be written separately instead of the averages. The name of the file is caller_script-YYYYMMDDTHHmmss.txt. If - is passed instead of a path, then STDOUT will be used instead. The timing stats are useful because the test harness doesn't normally measure time per subtest.

  • extend_stats (optional)

    This is to allow default output of stats_output_path to be fixed/reliable and anything additional in future versions requires extend_stats to be shown. Currently starting date/time in ISO_8601 is added.

USAGE NOTES

Not all tests can be modified to run under the aggregator, it is not intended for tests that require an isolated environment. So, for those that do not and can potentially run under the aggregator, sometimes very simple changes might be needed like giving unique names to subs (or not warning for redefines), replacing things that complain, restoring the environment at the end of the test etc.

The environment variable AGGREGATE_TESTS will be set while the tests are running. Example usage is a module that can only be loaded once, so you load it on the aggregated test file and then use something like this in the individual test files:

eval 'use My::Module' unless $ENV{AGGREGATE_TESTS};

Trying to aggregate too many tests into a single one can be counter-intuitive as you would ideally want to parallelize your test suite (so a super-long test continuing after the rest are done will slow down the suite). And in general more tests will run aggregated if they are grouped so that tests that can't be aggregated together are in different groups.

AUTHOR

Dimitrios Kechagias, <dkechag at cpan.org>

BUGS

Please report any bugs or feature requests to bug-test2-aggregate at rt.cpan.org, or through the web interface at https://rt.cpan.org/NoAuth/ReportBug.html?Queue=Test2-Aggregate. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

GIT

https://github.com/SpareRoom/Test2-Aggregate

COPYRIGHT & LICENSE

Copyright (C) 2019, SpareRoom.com

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.