Provided by: libtest-aggregate-perl_0.372-2_all bug


       Test::Aggregate - Aggregate "*.t" tests to make them run faster.


       Version 0.372


           use Test::Aggregate;

           my $tests = Test::Aggregate->new( {
               dirs => $aggregate_test_dir,
           } );

           ok $some_data, 'Test::Aggregate also re-exports Test::More functions';


       WARNING:  this is ALPHA code.  The interface is not guaranteed to be stable.  Further,
       check out Test::Aggregate::Nested (included with this distribution).  It's a more robust
       implementation which does not have the same limitations as "Test::Aggregate".

       A common problem with many test suites is that they can take a long time to run.  The
       longer they run, the less likely you are to run the tests.  This module borrows a trick
       from "Apache::Registry" to load up your tests at once, create a separate package for each
       test and wraps each package in a method named "run_the_tests".  This allows us to load
       perl only once and related modules only once.  If you have modules which are expensive to
       load, this can dramatically speed up a test suite.


       For a whole variety of reasons, tests run in BEGIN/CHECK/INIT/INIT blocks are now
       deprecated.  They cause all sorts of test sequence headaches.  Plus, they break the up-
       coming nested TAP work.  You will have a problem if you use this common idiom:

        BEGIN {
            use_ok 'My::Module' or die;

       Instead, just "use" the module and put the "use_ok" tests in a t/load.t file or something
       similar and don't aggregate it.  See the following for more information:


       Create a separate directory for your tests.  This should not be a subdirectory of your
       regular test directory.  Write a small driver program and put it in your regular test
       directory ("t/" is the standard):

        use Test::Aggregate;
        my $other_test_dir = 'aggregate_tests';
        my $tests = Test::Aggregate->new( {
           dirs => $other_test_dir

        ok $some_data, 'Test::Aggregate also re-exports Test::More functions';

       Take your simplest tests and move them, one by one, into the new test directory and keep
       running the "Test::Aggregate" program.  You'll find some tests will not run in a shared
       environment like this.  You can either fix the tests or simply leave them in your regular
       test directory.  See how this distribution's tests are organized for an example.

       Note that "Test::Aggregate" also exports all exported functions from "Test::More",
       allowing you to run other tests after the aggregated tests have run.

        use Test::Aggregate;
        my $other_test_dir = 'aggregate_tests';
        my $tests = Test::Aggregate->new( {
           dirs => $other_test_dir
        ok !(-f 't/data/tmp.txt'), '... and our temp file should be deleted';

       Some tests cannot run in an aggregate environment.  These may include test for this with
       the $ENV{TEST_AGGREGATE} variable:

        package Some::Package;

        BEGIN {
            die __PACKAGE__ ." cannot run in aggregated tests"
              if $ENV{TEST_AGGREGATE};


        my $tests = Test::Aggregate->new(
                dirs            => 'aggtests',
                verbose         => 1,            # optional, but recommended
                dump            => 'dump.t',     # optional
                shuffle         => 1,            # optional
                matching        => qr/customer/, # optional
                set_filenames   => 0,            # optional and not recommended
                tidy            => 1,            # optional and experimental
                test_nowarnings => 0,            # optional and experimental

       Creates a new "Test::Aggregate" instance.  Accepts a hashref with the following keys:

       ·   "dirs" (either this or "tests" is mandatory)

           The directories to look in for the aggregated tests.  This may be a scalar value of a
           single directory or an array refernce of multiple directories.

       ·   "tests" (either this or "dirs" is mandatory)

           Instead of providing directories for the aggregated tests, you may supply an array
           reference with a list of tests to aggregate.  If both are supplied, these tests will
           be appended to the list of tests found in "dirs".

           The "matching" parameter does not apply to test files identified with this key.

       ·   "verbose" (optional, but strongly recommended)

           If set with a true value, each test programs success or failure will be indicated with
           a diagnostic output.  The output below means that "aggtests/slow_load.t" was an
           aggregated test which failed.  This means it's much easier to determine which
           aggregated tests are causing problems.

            #     ok - aggtests/boilerplate.t
            #     ok - aggtests/00-load.t
            # not ok - aggtests/subs.t
            #     ok - aggtests/slow_load.t

           Note that three possible values are allowed for "verbose":

           ·   0 (default)

               No individual test program success or failure will be displayed.

           ·   1

               Only failing test programs will have their failure status shown.

           ·   2

               All test programs will have their success/failure shown.

       ·   "dump" (optional)

           You may list the name of a file to dump the aggregated tests to.  This is useful if
           you have test failures and need to debug why the tests failed.

       ·   "shuffle" (optional)

           Ordinarily, the tests are sorted by name and run in that order. This allows you to run
           them in any order.

       ·   "matching" (optional)

           If supplied with a regular expression (requires the "qr" operator), will only run
           tests whose filename matches the regular expression.

       ·   "set_filenames" (optional)

           If supplied with a true value, this will cause the following to be added for each

             local $0 = $test_filename;

           This is the default behavior.

       ·   "findbin" (optional)

           If supplied with a true value, this will cause FindBin::again() to be called before
           each test file.

           This is turned off by default.

           Note that older versions of FindBin (pre 1.47) sometimes get confused about where the
           bin directory is when I set $0.  I don't know why, but this is a rarely used option
           and only happens pre 5.8 perl, so I'm not too worried about it.  Just keep it in mind.

       ·   "dry" (optional)

           Just print the tests which will be run and the order they will be run in (obviously
           the order will be random if "shuffle" is true).

       ·   "tidy"

           If supplied a true value, attempts to run "Perl::Tidy" on the source code.  This is a
           no-op if "Perl::Tidy" cannot be loaded.  This option is "experimental".  Plus, if your
           tests are terribly convoluted, this could be slow and possibly buggy.

           If the value of this argument is the name of a file, assumes that this file is a
           ".perltidyrc" file.

       ·   "test_nowarnings"

           Disables "Test::NoWarnings" (fails if the module cannot be loaded).

           This is experimental and somewhat problematic.  Let me know if there are any problems.


       Attempts to aggregate and run all tests listed in the directories specified in the


       Since "BEGIN" and "END" blocks are for the entire aggregated tests and not for each test
       program (see "CAVEATS"), you might find that you need to have setup/teardown functions for
       tests.  These are useful if you need to setup connections to test databases, clear out
       temp files, or any of a variety of tasks that your test suite might require.  Here's a
       somewhat useless example, pulled from our tests:


        use strict;
        use warnings;

        use lib 'lib', 't/lib';
        use Test::Aggregate;
        use Test::More;

        my $dump = 'dump.t';

        my ( $startup, $shutdown ) = ( 0, 0 );
        my ( $setup,   $teardown ) = ( 0, 0 );
        my $tests = Test::Aggregate->new(
                dirs     => 'aggtests',
                dump     => $dump,
                startup  => sub { $startup++ },
                shutdown => sub { $shutdown++ },
                setup    => sub { $setup++ },
                teardown => sub { $teardown++ },
        is $startup,  1, 'Startup should be called once';
        is $shutdown, 1, '... as should shutdown';
        is $setup,    4, 'Setup should be called once for each test program';
        is $teardown, 4, '... as should teardown';

       Note that you can still dump these to a dump file.  This will only work if
       "Data::Dump::Streamer" 1.11 or later is installed.

       There are four attributes which can be passed to the constructor, each of which expects a
       code reference:

       ·   "startup"

            startup => \&connect_to_database,

           This function will be called before any of the tests are run.  It is not run in a
           BEGIN block.

       ·   "shutdown"

            shutdown => \&clean_up_temp_files,

           This function will be called after all of the tests are run.  It will not be called in
           an END block.

       ·   "setup"

            setup => sub {
               my $filename = shift;
               # this gets run before each test program.

           The setup function will be run before every test program.  The name of the test file
           will be passed as the first argument.

       ·   "teardown"

            teardown => sub {
               my $filename = shift;
               # this gets run after every test program.

           The teardown function gets run after every test program.  The name of the test file
           will be passed as the first argument.


       You shouldn't be using global variables and a dependence on them can break your code.
       However, Perl provides quite a few handy global variables which, unfortunately, can easily
       break your tests if you change them in one test and another assumes an unchanged value.
       As a result, we localize many of Perl's most common global variables for you, using the
       following syntax:

           local %ENV = %ENV;

       The following global variables are localized for you.  Any others must be localized
       manually per test.

       ·   @INC

       ·   %ENV

       ·   %SIG

       ·   $/

       ·   $_

       ·   $|


       Not all tests can be included with this technique.  If you have "Test::Class" tests, there
       is no need to run them with this.  Otherwise:

       ·   "exit"

           Don't call exit() in your aggregated tests.  We now warn very verbosely if this is
           done, but we still exit on the assumption that further tests cannot run.

       ·   "__END__" and "__DATA__" tokens.

           These won't work and the tests will call BAIL_OUT() if these tokens are seen.
           However, this limitation does not apply to Test::Aggregate::Nested.

       ·   "BEGIN" and "END" blocks.

           Since all of the tests are aggregated together, "BEGIN" and "END" blocks will be for
           the scope of the entire set of aggregated tests.  If you need setup/teardown
           facilities, see "TEARDOWN" in SETUP.

       ·   Syntax errors

           Any syntax errors encountered will cause this program to BAIL_OUT().  This is why it's
           recommended that you move your tests into your new directory one at a time:  it makes
           it easier to figure out which one has caused the problem.

       ·   "no_plan"

           Unfortunately, due to how this works, the plan is always "no_plan".
           for more information.

       ·   "Test::NoWarnings"

           Great module.  It loves to break aggregated tests since some might have warnings when
           others will not.  You can disable it like this:

            my $tests = Test::Aggregate->new(
                dirs    => 'aggtests/',
                startup => sub { $INC{'Test/'} = 1 },

           As an alternative, you can also disable it with:

            my $tests = Test::Aggregate->new({
               dirs            => 'aggtests',
               test_nowarnings => 0,

           We do work internally to subtract the extra test added by "Test::NoWarnings".  It's
           painful and experimental.  Good luck.

       ·   "Variable "$x" will not stay shared at (eval ..."

           Because each test is wrapped in a method call, any of your subs which access a
           variable in an outer scope will likely throw the above warning.  Pass in arguments
           explicitly to suppress this.

           Instead of:

            my $x = 17;
            sub foo {
                my $y = shift;
                return $y + $x;

           Write this:

            my $x = 17;
            sub foo {
                my ( $y, $x ) = @_;
                return $y + $x;

           However, consider Test::Aggregate::Nested.  This warning does not apply with that

       ·   Singletons

           Be very careful of code which loads singletons.  Oftimes those singletons in test
           suites may be altered for testing purposes, but later attempts to use those singletons
           can fail dramatically as they're not expecting the alterations.  (Your author has
           painfully learned this lesson with database connections).


       Before aggregating tests, make sure that you add tests one at a time to the aggregated
       test directory.  Attempting to add many tests to the directory at once and then
       experiencing a failure means it will be much harder to track down which tests caused the

       Debugging aggregated tests which fail is a multi-step process.  Let's say the following

        my $tests = Test::Aggregate->new(
                dump    => 'dump.t',
                shuffle => 1,
                dirs    => 'aggtests',

   Manually run the tests
       The first step is to manually run all of the tests in the "aggtests" dir.

        prove -r aggtests/

       If the failures appear the same, fix them just like you would fix any other test failure
       and then rerun the "Test::Aggregate" code.

       Sometimes this means that a different number of tests run from what the aggregted tests
       run.  Look for code which ends the program prematurely, such as an exception or an "exit"

   Run a dump file
       If this does not fix your problem, create a dump file by passing "dump => $dumpfile" to
       the constructor (as in the above example).  Then try running this dumpfile directly to
       attempt to replicate the error:

        prove -r $dumpfile

   Tweaking the dump file
       Assuming the error has been replicated, open up the dump file.  The beginning of the dump
       file will have some code which overrides some "Test::Builder" internals.  After that,
       you'll see the code which runs the tests.  It will look similar to this:

        if ( __FILE__ eq 'dump.t' ) {
            Test::More::diag("******** running tests for aggtests/boilerplate.t ********")
               if $ENV{TEST_VERBOSE};

            Test::More::diag("******** running tests for aggtests/subs.t ********")
               if $ENV{TEST_VERBOSE};

            Test::More::diag("******** running tests for aggtests/00-load.t ********")
               if $ENV{TEST_VERBOSE};

            Test::More::diag("******** running tests for aggtests/slow_load.t ********")
               if $ENV{TEST_VERBOSE};

       You can try to narrow down the problem by commenting out all of the "run_the_tests" lines
       and gradually reintroducing them until you can figure out which one is actually causing
       the failure.


   My Tests Threw an Exception But Passed Anyway!
       This really isn't a "Test::Aggregate" problem so much as a general Perl problem.  For each
       test file, "Test::Aggregate" wraps the tests in an eval and checks "my $error = $@".
       Unfortunately, we sometimes get code like this:


       And internally, the 'Server' class throws an exception but uses its own evals in a
       "DESTROY" block (or something similar) to trap it.  If the code you call uses an eval but
       fails to localize it, it wipes out your eval.  Neat, eh?  Thus, you never get a chance to
       see the error.  For various reasons, this tends to impact "Test::Aggregate" when a
       "DESTROY" block is triggered and calls code which internally uses eval (e.g.,
       "DBIx::Class").  You can often fix this with:

        DESTROY {
           local $@ = $@;  # localize but preserve the value
           my $self = shift;
           # do whatever you want

   "BEGIN" and "END" blocks
       Remember that since the tests are now being run at once, these blocks will no longer run
       on a per-test basis, but will run for the entire aggregated set of tests.  You may need to
       examine these individually to determine the problem.

   "CHECK" and "INIT" blocks.
       Sorry, but you can't use these (just as in modperl).  See perlmod for more information
       about them and why they won't work.

       This is a great test module.  When aggregating tests together, however, it can cause pain
       as you'll often discover warnings that you never new existed.  For a quick fix, add this
       before you attempt to run your tests:

        $INC{'Test/'} = 1;

       That will disable "Test::NoWarnings", but you'll want to go in later to fix them.

       Many tests make assumptions about the paths to files and moving them into a new test
       directory can break this.

       Tests which use $0 can be problematic as the code is run in an "eval" through
       "Test::Aggregate" and $0 may not match expectations.  This also means that it can behave
       differently if run directly from a dump file.

       As it turns out, you can assign to $0!  We do this by default and set the $0 to the
       correct filename.  If you don't want this behavior, pass "set_filenames => 0" to the

   Minimal test case
       If you cannot solve the problem, feel free to try and create a minimal test case and send
       it to me (assuming it's something I can run).


       Curtis Poe, "<ovid at>"


       Please report any bugs or feature requests to "bug-test-aggregate at", or
       through the web interface at
       <>.  I will be notified, and
       then you'll automatically be notified of progress on your bug as I make changes.


       You can find documentation for this module with the perldoc command.

           perldoc Test::Aggregate

       You can also find information oneline:



       Many thanks to mauzo (<> for helping me find the 'skip_all'

       Thanks to Johan Lindstroem for pointing me to Apache::Registry.


       Copyright 2007 Curtis "Ovid" Poe, all rights reserved.

       This program is free software; you can redistribute it and/or modify it under the same
       terms as Perl itself.