NAME

bisect.pl - use git bisect to pinpoint changes

SYNOPSIS

# When did this become an error?
.../Porting/bisect.pl -e 'my $a := 2;'
# When did this stop being an error?
.../Porting/bisect.pl --expect-fail -e '1 // 2'
# When did this test start failing?
.../Porting/bisect.pl --target t/op/sort.t
# When were all lines matching this pattern removed from all files?
.../Porting/bisect.pl --match '\b(?:PL_)hash_seed_set\b'
# When was some line matching this pattern added to some file?
.../Porting/bisect.pl --expect-fail --match '\buseithreads\b'
# When did this test program stop exiting 0?
.../Porting/bisect.pl -- ./perl -Ilib ../test_prog.pl
# When did this test program start crashing (any signal or coredump)?
.../Porting/bisect.pl --crash -- ./perl -Ilib ../test_prog.pl
# When did this first become valid syntax?
.../Porting/bisect.pl --target=miniperl --end=v5.10.0 \
     --expect-fail -e 'my $a := 2;'
# What was the last revision to build with these options?
.../Porting/bisect.pl --test-build -Dd_dosuid
# When did this test program start generating errors from valgrind?
.../Porting/bisect.pl --valgrind ../test_prog.pl
# When did these cpan modules start failing to compile/pass tests?
.../Porting/bisect.pl --module=autobox,Moose
# When did this code stop working in blead with these modules?
.../Porting/bisect.pl --with-module=Moose,Moo -e 'use Moose; 1;'
# Like the above 2 but with custom CPAN::MyConfig
.../Porting/bisect.pl --module=Moo --cpan-config-dir=/home/blah/custom/

DESCRIPTION

Together bisect.pl and bisect-runner.pl attempt to automate the use of git bisect as much as possible. With one command (and no other files) it's easy to find out

• Which commit caused this example code to break?

• Which commit caused this example code to start working?

• Which commit added the first file to match this regex?

• Which commit removed the last file to match this regex?

usually without needing to know which versions of perl to use as start and end revisions.

By default bisect.pl will process all options, then use the rest of the command line as arguments to list system to run a test case. By default, the test case should pass (exit with 0) on earlier perls, and fail (exit non-zero) on blead. bisect.pl will use bisect-runner.pl to find the earliest stable perl version on which the test case passes, check that it fails on blead, and then use bisect-runner.pl with git bisect run to find the commit which caused the failure.

Many of perl's own test scripts exit 0 even if their TAP reports test failures, and some need particular setup (such as running from the right directory, or adding -T to the command line). Hence if you want to bisect a test script, you can specify it with the --target option, and it will be invoked using t/TEST which performs all the setup, and exits non-zero if the TAP reports failures. This works for any file ending .t, so you can use it with a file outside of the working checkout, for example to test a particular version of a test script, as a path inside the repository will (of course) be testing the version of the script checked out for the current revision, which may be too early to have the test you are interested in.

Because the test case is the complete argument to system, it is easy to run something other than the perl built, if necessary. If you need to run the perl built, you'll probably need to invoke it as ./perl -Ilib .... As a special case, if the first argument of the test case is a readable file (whether executable or not), matching qr{\A#!./(?:mini)?perl\b} then it will have ./perl <-Ilib> (or ./miniperl) prepended to it.

You need a clean checkout to run a bisect. You can use the checkout containing Porting/bisect.pl if you wish - in this case Porting/bisect.pl will copy Porting/bisect-runner.pl to a temporary file generated by File::Temp::tempfile(). If doing this, beware that when the bisect ends (or you abort it) then your checkout is no longer at blead, so you will need to git checkout blead before restarting, to get the current version of Porting/bisect.pl again. It's often easier either to copy Porting/bisect.pl and Porting/bisect-runner.pl to another directory (e.g. ~/bin, if you have one), or to create a second git repository for running bisect. To create a second local repository, if your working checkout is called perl, a simple solution is to make a local clone, and run from that. i.e.:

cd ..
git clone perl perl2
cd perl2
../perl/Porting/bisect.pl ...

By default, bisect-runner.pl will automatically disable the build of DB_File for commits earlier than ccb44e3bf3be2c30, as it's not practical to patch DB_File 1.70 and earlier to build with current Berkeley DB headers. (ccb44e3bf3be2c30 was in September 1999, between 5.005_62 and 5.005_63.) If your db.h is old enough you can override this with -Unoextensions.

OPTIONS

• --start commit-ish

  Earliest revision to test, as a commit-ish (a tag, commit or anything else git understands as a revision). If not specified, bisect.pl will search stable .0 perl releases until it finds one where the test case passes. The default is to search from 5.002 to the most recent tagged stable release (v5.18.0 at the time of writing). If bisect.pl detects that the checkout is on a case insensitive file system, it will search from 5.005 to the most recent tagged stable release. Only .0 stable releases are used because these are the only stable releases that are parents of blead, and hence suitable for a bisect run.

• --end commit-ish

  Most recent revision to test, as a commit-ish. If not specified, defaults to blead.

• --target target

  Makefile target (or equivalent) needed, to run the test case. If specified, this should be one of

  • none

    Don't build anything - just run the user test case against a clean checkout. Using this gives a couple of features that a plain git bisect run can't offer - automatic start revision detection, and test case --timeout.

  • config.sh

    Just run ./Configure

  • config.h

    Run the various *.SH files to generate Makefile, config.h, etc.

  • miniperl

    Build miniperl.

  • lib/Config.pm

    Use miniperl to build lib/Config.pm

  • Fcntl

    Build lib/auto/Fcntl/Fnctl.so (strictly, .$Config{so}). As Fcntl is simple XS module present since 5.000, this provides a fast test of whether XS modules can be built. Note, XS modules are built by miniperl, hence this target will not build perl.

  • perl

    Build perl. This also builds pure-Perl modules in cpan, dist and ext. XS modules (such as Fcntl) are not built.

  • test_prep

    Build everything needed to run the tests. This is the default if we're running test code, but is time consuming, as it means building all XS modules. For older Makefiles, the previous name of test-prep is automatically substituted. For very old Makefiles, make test is run, as there is no target provided to just get things ready, and for 5.004 and earlier the tests run very quickly.

  • A file ending .t

    Build everything needed to run the tests, and then run this test script using t/TEST. This is actually implemented internally by using the target test_prep, and setting the test case to "sh", "-c", "cd t && ./TEST ..."

• --one-liner 'code to run'

• -e 'code to run'

  Example code to run, just like you'd use with perl -e.

  This prepends ./perl -Ilib -e 'code to run' to the test case given, or ./miniperl if target is miniperl.

  (Usually you'll use -e instead of providing a test case in the non-option arguments to bisect.pl. You can repeat -e on the command line, just like you can with perl)

  -E intentionally isn't supported, as it's an error in 5.8.0 and earlier, which interferes with detecting errors in the example code itself.

• -c

  Add -c to the command line, to cause perl to exit after syntax checking.

• -l

  Add -l to the command line with -e

  This will automatically append a newline to every output line of your testcase. Note that you can't specify an argument to perl's -l with this, as it's not feasible to emulate perl's somewhat quirky switch parsing with Getopt::Long. If you need the full flexibility of -l, you need to write a full test case, instead of using bisect.pl's -e shortcut.

• -w

  Add -w to the command line with -e

  It's not valid to pass -c, -l or -w to bisect.pl unless you are also using -e

• --expect-fail

  The test case should fail for the start revision, and pass for the end revision. The bisect run will find the first commit where it passes.

• --crash

  Treat any non-crash as success, any crash as failure. (Crashing defined as exiting with a signal or a core dump.)

• -D config_arg=value

• -U config_arg

• -A config_arg=value

  Arguments (-A, -D, -U) to pass to Configure. The -D, -A and -U switches should be spelled as if you were normally giving them to ./Configure. For example,

  -Dnoextensions=Encode
  -Uusedevel
  -Accflags=-DNO_MATHOMS

  Repeated -A arguments are passed through as is. -D and -U are processed in order, and override previous settings for the same parameter. bisect-runner.pl emulates -Dnoextensions when Configure itself does not provide it, as it's often very useful to be able to disable some XS extensions.

• --make make-prog

  The make command to use. If this not set, make is used. If this is set, it also adds a -Dmake=... else some recursive make invocations in extensions may fail. Typically one would use this as --make gmake to use gmake in place of the system make.

• --jobs jobs

• -j jobs

  Number of make jobs to run in parallel. A value of 0 suppresses parallelism. If /proc/cpuinfo exists and can be parsed, or /sbin/sysctl exists and reports hw.ncpu, or /usr/bin/getconf exists and reports _NPROCESSORS_ONLN defaults to 1 + number of CPUs. On HP-UX with the system make defaults to 0, otherwise defaults to 2.

• --match pattern

• --no-match pattern

  Instead of running a test program to determine pass or fail, --match will pass if the given regex matches, and hence search for the commit that removes the last matching file. --no-match inverts the test, to search for the first commit that adds files that match.

  The remaining command line arguments are treated as glob patterns for files to match against. If none are specified, then they default as follows:

  • If no target is specified, the match is against all files in the repository (which is fast).

  • If a target is specified, that target is built, and the match is against only the built files.

  Treating the command line arguments as glob patterns should not cause problems, as the perl distribution has never shipped or built files with names that contain characters which are globbing metacharacters.

  Anything which is not a readable file is ignored, instead of generating an error. (If you want an error, run grep or ack as a test case). This permits one to easily search in a file that changed its name. For example:

  .../Porting/bisect.pl --match 'Pod.*Functions' 'pod/buildtoc*'

  --no-match ... is implemented as --expect-fail --match ...

• --valgrind

  Run the test program under valgrind. If you need to test for memory errors when parsing invalid programs, the default parser fail exit code of 255 will always override valgrind, so try putting the test case invalid code inside a string eval, so that the perl interpreter will exit with 0. (Be sure to check the output of $@, to avoid missing mistakes such as unintended eval failures due to incorrect @INC)

  Specifically, this option prepends valgrind --error-exitcode=124 to the command line that runs the testcase, to cause valgrind to exit non-zero if it detects errors, with the assumption that the test program itself always exits with zero. If you require more flexibility than this, either specify your valgrind invocation explicitly as part of the test case, or use a wrapper script to control the command line or massage the exit codes.

  In order for the test program to be seen as a perl script to valgrind (rather than a shell script), the first line must be one of the following

  #!./perl
  #!./miniperl

• --test-build

  Test that the build completes, without running any test case.

  By default, if the build for the desired target fails to complete, bisect-runner.pl reports a skip back to git bisect, the assumption being that one wants to find a commit which changed state "builds && passes" to "builds && fails". If instead one is interested in which commit broke the build (possibly for particular Configure options), use --test-build to treat a build failure as a failure, not a "skip".

  Often this option isn't as useful as it first seems, because any build failure will be reported to git bisect as a failure, not just the failure that you're interested in. Generally, to debug a particular problem, it's more useful to use a target that builds properly at the point of interest, and then a test case that runs make. For example:

  .../Porting/bisect.pl --start=perl-5.000 --end=perl-5.002 \
      --expect-fail --force-manifest --target=miniperl make perl

  will find the first revision capable of building DynaLoader and then perl, without becoming confused by revisions where miniperl won't even link.

• --module module1,module2,...

  Install this (or these) module(s), die when it (the last of those) cannot be updated to the current version.

  Misnomer. the argument can be any argument that can be passed to CPAN shell's install command. But: since we only have the uptodate command to verify that an install has taken place, we are unable to determine success for arguments like MSCHWERN/Test-Simple-1.005000_005.tar.gz.

  In so far, it is not such a misnomer.

  Note that this and --with-module will both require a CPAN::MyConfig. If $ENV{HOME}/.cpan/CPAN/MyConfig.pm does not exist, a CPAN shell will be started up for you so you can configure one. Feel free to let CPAN pick defaults for you. Enter 'quit' when you are done, and then everything should be all set. Alternatively, you may specify a custom CPAN::MyConfig by using --cpan-config-dir.

  Also, if you want to bisect a module that needs a display (like TK) and you don't want random screens appearing and disappearing on your computer while you're working, you can do something like this:

  In a terminal:

  $ while true; do date ; if ! ps auxww | grep -v grep \
    | grep -q Xvfb; then Xvfb :121 & fi; echo -n 'sleeping 60 '; \
    sleep 60; done

  And then:

  DISPLAY=":121" .../Porting/bisect.pl --module=TK

  (Some display alternatives are vncserver and Xnest.)

• --with-module module1,module2,...

  Like --module above, except this simply installs the requested modules and they can then be used in other tests.

  For example:

  .../Porting/bisect.pl --with-module=Moose -e 'use Moose; ...'

• --no-module-tests

  Use in conjunction with --with-module to install the modules without running their tests. This can be a big time saver.

  For example:

  .../Porting/bisect.pl --with-module=Moose --no-module-tests \
       -e 'use Moose; ...'

• --test-module

  This is like --module, but just runs the module's tests, instead of installing it.

  WARNING: This is a somewhat experimental option, known to work on recent CPAN shell versions. If you use this option and strange things happen, please report them.

  Usually, you can just use --module, but if you are getting inconsistent installation failures and you just want to see when the tests started failing, you might find this option useful.

• --cpan-config-dir /home/blah/custom

  If defined, this will cause CPAN to look for CPAN/MyConfig.pm inside of the specified directory, instead of using the default config of $ENV{HOME}/.cpan/.

  If no default config exists, a CPAN shell will be fired up for you to configure things. Letting CPAN automatically configure things for you should work well enough. You probably want to choose manual instead of local::lib if it asks. When you're finished with configuration, just type q and hit ENTER and the bisect should continue.

• --force-manifest

  By default, a build will "skip" if any files listed in MANIFEST are not present. Usually this is useful, as it avoids false-failures. However, there are some long ranges of commits where listed files are missing, which can cause a bisect to abort because all that remain are skipped revisions.

  In these cases, particularly if the test case uses miniperl and no modules, it may be more useful to force the build to continue, even if files MANIFEST are missing.

• --force-regen

  Run make regen_headers before building miniperl. This may fix a build that otherwise would skip because the generated headers at that revision are stale. It's not the default because it conceals this error in the true state of such revisions.

• --expect-pass [0|1]

  --expect-pass=0 is equivalent to --expect-fail. 1 is the default.

• --timeout seconds

  Run the testcase with the given timeout. If this is exceeded, kill it (and by default all its children), and treat it as a failure.

• --setpgrp

  Run the testcase in its own process group. Specifically, call setpgrp 0, 0 just before exec-ing the user testcase. The default is not to set the process group, unless a timeout is used.

• --all-fixups

  bisect-runner.pl will minimally patch various files on a platform and version dependent basis to get the build to complete. Normally it defers doing this as long as possible - .SH files aren't patched until after Configure is run, and C and XS code isn't patched until after miniperl is built. If --all-fixups is specified, all the fixups are done before running Configure. In rare cases adding this may cause a bisect to abort, because an inapplicable patch or other fixup is attempted for a revision which would usually have already skipped. If this happens, please report it as a bug, giving the OS and problem revision.

• --early-fixup file

• --late-fixup file

  Specify a file containing a patch or other fixup for the source code. The action to take depends on the first line of the fixup file

  • #!perl

    If the first line starts #!perl then the file is run using $^X

  • #!/absolute/path

    If a shebang line is present the file is executed using system

  • filename =~ /pattern/

  • filename !~ /pattern/

    If filename does not exist then the fixup file's contents are ignored. Otherwise, for =~, if it contains a line matching pattern, then the file is fed to patch -p1 on standard input. For =~, the patch is applied if no lines match the pattern.

    As the empty pattern in Perl is a special case (it matches the most recent successful match) which is not useful here, the treatment of an empty pattern is special-cased. filename =~ // applies the patch if filename is present. filename !~ // applies the patch if filename missing. This makes it easy to unconditionally apply patches to files, and to use a patch as a way of creating a new file.

  • Otherwise, the file is assumed to be a patch, and always applied.

  early-fixups are applied before ./Configure is run. late-fixups are applied just after ./Configure is run.

  These options can be specified more than once. file is actually expanded as a glob pattern. Globs that do not match are errors, as are missing files.

• --no-clean

  Tell bisect-runner.pl not to clean up after the build. This allows one to use bisect-runner.pl to build the current particular perl revision for interactive testing, or for debugging bisect-runner.pl.

  Passing this to bisect.pl will likely cause the bisect to fail badly.

• --validate

  Test that all stable (.0) revisions can be built. By default, attempts to build blead, then tagged stable releases in reverse order down to perl-5.002 (or perl5.005 on a case insensitive file system). Stops at the first failure, without cleaning the checkout. Use --start to specify the earliest revision to test, --end to specify the most recent. Useful for validating a new OS/CPU/compiler combination. For example

  ../perl/Porting/bisect.pl --validate -le 'print "Hello from $]"'

  If no testcase is specified, the default is to use t/TEST to run t/base/*.t

• --check-args

  Validate the options and arguments, and exit silently if they are valid.

• --check-shebang

  Validate that the test case isn't an executable file with a #!/usr/bin/perl line (or similar). As bisect-runner.pl does not automatically prepend ./perl to the test case, a #! line specifying an external perl binary will cause the test case to always run with that perl, not the perl built by the bisect runner. Likely this is not what you wanted. If your test case is actually a wrapper script to run other commands, you should run it with an explicit interpreter, to be clear. For example, instead of ../perl/Porting/bisect.pl ~/test/testcase.pl you'd run ../perl/Porting/bisect.pl /usr/bin/perl ~/test/testcase.pl

• --gold

  Revision to use when checking out known-good recent versions of files, such as makedepend.SH. bisect-runner.pl defaults this to blead, but bisect.pl will default it to the most recent stable release.

• --usage

• --help

• -?

  Display the usage information and exit.

EXAMPLES

Code has started to crash under miniperl

• Problem

  Under make minitest (but not under make test_harness), t/re/pat.t was failing to compile. What was the first commit at which that compilation failure could be observed?

• Solution

  Extract code from the test file at the point where ./miniperl -Ilib -c was showing a compilation failure. Use that in bisection with the miniperl target.

  .../Porting/bisect.pl --target=miniperl --start=2ec4590e \
      -e 'q|ace| =~ /c(?=.$)/; $#{^CAPTURE} == -1); exit 0;'

• Reference

  GH issue 17293

Blead breaks CPAN on threaded builds only

• Problem

  Tests in CPAN module XML::Parser's test suite had begun to fail when tested against blead in threaded builds only.

• Solution

  Provide Configure-style switch to bisection program. Straightforward use of the --module switch.

  .../Porting/bisect.pl -Duseithreads \
      --start=6256cf2c \
      --end=f6f85064 \
      --module=XML::Parser

• Reference

  GH issue 16918

Point in time where code started to segfault is unknown

• Problem

  User submitted code sample which when run caused perl to segfault, but did not claim that this was a recent change.

• Solution

  Used locally installed production releases of perl (previously created by perlbrew) to identify the first production release at which the code would not compile. Used that information to shorten bisection time.

  .../perl Porting/bisect.pl \
      --start=v5.14.4 \
      --end=v5.16.3 \
      --crash -- ./perl -Ilib /tmp/gh-17333-map.pl
  
  $ cat gh-17333-map.pl
  
  @N = 1..5;
  map { pop @N } @N;

• Reference

  GH issue 17333

When did perl start failing to build on a certain platform using g++ as the C-compiler?

• Problem

  On NetBSD-8.0, perl had never been smoke-tested using g++ as the C-compiler. Once this was done, it became evident that changes in that version of the operating system's code were incompatible with some perl source written long before that OS version was ever released!

• Solution

  Bisection range was first narrowed using existing builds at release tags. Then, bisection specified the C-compiler via Configure-style switch and used --test-build to identify the commit which "broke" the build.

  .../perl Porting/bisect.pl \
      -Dcc=g++ \
      --test-build \
      --start=v5.21.6 \
      --end=v5.21.7

  Then, problem was discussed with knowledgeable NetBSD user.

• Reference

  GH issue 17381