NAME

App::TestOnTap::Args

Commandline options and arguments understood for TestOnTap processing

SYNOPSIS

testontap
	[ @argfile ]      (see 'EXPANDING FILES IN ARGUMENT LIST' below for details) 
   [
       --usage | -?
           |
       --help | -h
           |
       --manual
           |
       --version
   ]
   [ -v | --verbose [ -v ... ] ]
   [ --workdirectory <path> ]
   [ { --skip || --include } <query> ]
   [ --jobs <n> ]
   [ --order <strategy> ]
   [ --timer || --no-timer ]
   [ --harness || --no-harness ]
   [ --dryrun || --no-dryrun ]
   [ --merge || --no-merge ]
   [ --configuration <cfgfile> || --cfg <cfgfile> ]
   [ --archive ]
   [ --savedirectory <path> ]
   [ --define | -D <key>=<value> [ --define ... ] ]
   <suitepath>
   [<suiteargs> ...]  

OPTIONS AND ARGUMENTS

All options can be abbreviated, as long as they are unambiguous. Option matching is case sensitive. Forcibly end option parsing using '--'.

EXPANDING FILES IN ARGUMENT LIST

The command line may at any place contain arguments of the form @filename, which means that the filename will be read for arguments to insert in place. This will help if the command line is very long or is just nicely persisted for reuse.

Further, this will work recursively, i.e. any resulting argument of the same form, will be read. If the filename is relative, it's assumed relative to the directory the previous file was in.

The example command line testontap --verbose @myargs --jobs 10 @moreargs will be expanded by reading the myargs and moreargs files (in the current directory, obviously). The read files may contain a line of @/some/path/to/globalargs, and assuming that file in turn contains @extraargs, that file is expected to exist as /some/path/to/extraargs.

Note that the file is line-oriented - each line in such a file will become a single argument. No whitespace splitting inside each value will occur, but some rules apply:

• Environment variable expansion

  Any construction of the form ${name} will be assumed to be a reference to the environment variable 'name', and the contents of it will be inserted in place. If the environment variable doesn't exist it is an error.

• Line continuation

  To avoid physical lines in the file to be extremely long, it can be continued on the next physical (to any length) by ending the line with a backslash (\).

  Example:
    fee \
    fie \
    foo

  Will become the argument 'fee fie foo'.

• Comments

  Any line (after completed line continuation and not counting inital whitespace) having a '#' as the first character, will be ignored as being a comment.

• Trimming

  Any beginning and ending whitespace will be trimmed away.

• Empty lines

  Any empty line will be ignored.

INFORMATION OPTIONS

If any of these options are given, the rest of the command line is ignored.

If multiple of these options are given only one will be acted on, in the order --manual, --help, --usage and --version.

--usage | -?

Displays basic usage information.

--help

Displays usage and help with options/arguments.

--manual

Displays the full manual page.

--version

Displays the script name and version information.

-v, --verbose

By default, the output will be regular test output after consuming TAP (e.g. timings, ok/fail messages, summary etc).

Using this option allows the pass through of more verbose output from the tests themselves.

To up verbosity, add one or more verbose flags (bundling for singlecharacter options is on, so '-vvv' is the same as '-v -v -v'). Currently, up to verbosity level 1 is used.

--workdirectory <path>

By default, a temporary directory is created to hold the data produced by the tests and is automatically removed when the run is complete.

Use this option if you want to keep the work dir after completion (for debugging/perusal/whatever). The path must not exist beforehand.

For details of the work directory, see the manual regarding 'persisted results'.

NOTE: by using this flag, the --merge option is automatically turned on.

--skip <query> or --include <query>

Note: the --skip and --include flags are mutually exclusive - simply choose the one that is easiest to express your intention with.

By default, all tests will be run (subject to selection by the execmap in effect and the internally configured skip).

By providing a query, a subset of those tests can be skipped/included. However, be aware that the actual tests selected are also subject to any dependencies declared between tests - e.g. a query skipping test Z will still select test Z if there is a Y => Z dependency in the config.

For more on selecting tests, also see the manual regarding the 'skip' config setting.

--jobs <n>

By default, tests will run in sequence, governed by any dependencies, and sorted alphabetically. Also by default, they will always run serially.

By setting this option to a value higher than 1 allowing a maximum of n jobs to execute concurrently. However, for this to take effect, a parallelizable rule must be in the suite configuration in order to select which tests actually can run in parallel

The actual amount of concurrently executing jobs are then subject to limiting factors, such as dependencies and the parallelizability of a given test.

For more on parallelization, see the manual regarding the 'parallelizable' config setting.

IMPORTANT NOTE: On Windows in particular, fork() is constrained in the number of parallel forks, so this value will be capped to a maximum of 60 at this point, while defaulting to effectively 'infinite' on other platforms. To experiment with other maximums, set the environment variable _TESTONTAP_MAX_JOBS to the desired number.

--order <strategy>

Setting a strategy controls the ordering of the tests.

However, note that the actual order of tests are primarily controlled by dependencies and possibly the parallelizability of any given test. Thus, the order strategy is only relevant when there are multiple tests to choose from during evaluation of 'next available test(s)'.

The default strategy is 'none', but the default for a given suite may also be set in its configuration.

• none

  This is the default strategy and the resulting order depends on internal implementation details. It may or may not look random, but for a guaranteed randomization, use the random strategy.

• alphabetic or ralphabetic

  This strategy simply uses an alphabetic sort on test names. Thus, this results in the order 'sm1', 'sm10', 'sm2' etc.

  Adding the initial r causes the order to be reversed.

• natural or rnatural

  This strategy makes a best effort to take numbering into account when sorting. It is basically alphabetic, but takes embedded numbers into account if possible. Thus, this results in the order 'sm1', 'sm2', 'sm10' etc.

  Adding the initial r causes the order to be reversed.

• random

  This will explicitly shuffle the order randomly.

--timer, --no-timer

Using --timer turns on the display of timing information for tests when running.

Note that the information is always present in persisted results, this just controls the display when running a suite.

--harness, --no-harness

By default test output is handled by a test harness.

Using --no-harness turns on a special mode where tests are run with no capture and parsing of output. Printouts go directly to stdout/err, and stdin is active.

This can be convenient to directly see the TAP (or other) output which can be handy for debugging the tests themselves. Also, since stdin should work normally, using an execmap that starts the test in debug mode enables them to run under a real debugger.

# Example config file 'debug.cfg':

# (copies of the real config settings)

execmap = debug normal

[EXECMAP normal]
match = regexp(\.pl$)
cmd = perl

[EXECMAP debug]
match = eq(some/test.pl)
cmd = perl -d

and run with:

testontap --no-harness --configuration debug.execmap some/path/to/suite

would start the some/test.pl file under the Perl debugger but run the rest normally.

Note that this renders several options and mechanisms useless: for example --verbose, --jobs, --timer. Also, the work directory will miss most of the data normally seen, such as TAP output, aggregated test results etc. --no-harness will be automatically used when --dryrun is used

--dryrun, --no-dryrun

By using --dryrun, --no-harness is implied.

No tests will be actually run, the tests will just be listed in the order they would have run in the --no-harness format plus lists of what any dependency queries actually select.

--merge, --no-merge

By default, writes to stderr in tests are passed directly through to the console (but see below).

However, as STDERR is unbuffered, this tends to make the console look very ugly and the output are only marginally in sync with the ordinary harness output.

Using --merge makes the harness merge stdout and stderr as one, which makes them synchronized. However, the output is no longer visible unless --verbose is used.

If any of the options --archive, --savedirectory or --workdirectory are given, merging is turned on, in order to force the output to the result 'tap' files. If this is not desired, use --no-merge explicitly.

--configuration <cfgfile> (or --cfg ...)

By default, the test suite is required to have a configuration file (even if it's empty). With this option, another configuration file can be used instead.

If this option is not used, and the configuration in the suite, some internal defaults will be used.

For more on configuration, see the manual regarding the config setting.

--archive, --savedirectory <path>

By default, the results of a test run are not saved.

By providing either or both of these options, results may be saved as either a ready-made archive or a regular directory. The name of the saved file/directory will be named using the basename of the suite plus a timestamp and the run id, which together should be entirely unique.

Giving only --archive, the result will be saved as an archive in the current directory.

Giving --archive --savedirectory some/path together, the result will be saved as an archive in the directory some/path (the directory may or may not exist beforehand).

Giving only --savedirectory some/path, the result archive will be a regular subdirectory in the directory some/path (the directory may or may not exist beforehand).

For details of saved results, see the manual regarding 'persisted results'.

NOTE: by using either of these flags, the --merge option is automatically turned on.

--define <key>=<value>

The function of this option is to make it possible to pass on arbitrary key/value pairs to a saved result in order to later enable various types of differentiation to be made when processing the saved result. The defines are not made available to tests.

Can be given multiple times.

suitepath

This is the path to the root directory of the suite, or to a zip archive containing a suite.

If it is a zip archive, the archive must contain only a directory at the zip root, and a valid testontap configuration file in that directory. It will be unpacked to a temporary location before being executed and cleaned up after.

For details on test suites, see the manual.

suiteargs ...

Any arguments can be passed, and they will just be passed on as-is to the tests when they start.

For details, see the manual on parameter passing.

MORE HELP

For full information on TestOnTap, see the manual, or use --manual or run perldoc App::TestOnTap to see the manual page.