NAME

       Lintian::Tutorial::TestSuite -- Quick intro to running the Lintian testsuite

SYNOPSIS

       This guide will quickly introduce you to running the Lintian test suite and some tricks.
       The Lintian test suite is fairly large and accordingly it can take a substantial amount of
       time to run.  To speed up development, there are various options to limit the tests to
       run.

       If you are looking for a guide on how to write tests, please consult
       Lintian::Tutorial::WritingTests.

DESCRIPTION

       The Lintian test suite is an extensive collection of various test cases.  The test suite
       is divided into 4 "sub-suites".  The majority of tests are currently located in the
       "tests" sub-suite.

       To run the full suite in all its glory, simply invoke:

        $ debian/rules runtests

        OR

        $ mkdir -p debian/test-out
        $ t/runtests -k --dump-logs t debian/test-out

       While writing a new tag (or check) you probably only want to run a particular (subset of
       the) test(s).  See "Running a subset of the tests" for the available options.

       When run via debian/rules, the test suite respects "DEB_BUILD_OPTIONS=parallel=N". When
       using t/runtests directly, use -jN to choose the number of threads.  Note that "N" denotes
       the amount of "worker" threads.  The test runner will generally have 2 threads more than
       that.  Also each "worker" will run lintian, which runs multiple unpacking jobs in parallel
       as well.

       In case you are wondering about the 2 extra threads in the test runner, the first of them
       is the "coordinator" thread (which will generally be waiting when the workers are active)
       and the second one is the "output" thread (which handles the fancy output).

   Running a subset of the tests
       The following options are available:

       Running a single test
           To run a single test by its name, use:

            $ debian/rules runtests onlyrun=$name

            OR

            $ t/runtests --dump-logs t debian/test-out $name

       Running all tests for a check
           To run all tests for a given check, use:

            $ debian/rules runtests onlyrun=$check

            OR

            $ t/runtests --dump-logs -k t debian/test-out $check

           $check must be the name of a check (it will test for checks/$check.desc) or "legacy".
           This will run all tests that start with "$check-".

           Note: The "changes" sub-suite in the new test suite does not support this.

       Running all tests in a sub-suite
           To run all tests in a given sub-suite, use:

            $ debian/rules runtests onlyrun=suite:$suites

            OR

            $ t/runtests --dump-logs -k t debian/test-out suite:$suites

           $suites is a comma-separated list of names of sub-suites to run.

           Note: this cannot be used to influence the order in which the sub-suites are run.

       Running all tests designed for a specific tag
           To run all tests that have a "Test-For" or a "Test-Against" for a given tag, use:

            $ debian/rules runtests onlyrun=tag:$tag

            OR

            $ t/runtests --dump-logs -k t debian/test-out tag:$tag

   Running tests under coverage
       It is possible to run most of the tests under Devel::Cover.  This is done by passing
       --coverage to t/runtests.  Example:

         $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out

       Please note that Devel::Cover does not seem to handle multiple threads too well.  You may
       see spurious warnings/errors if you run the tests with 2 or more active worker threads.

       Caveat 1: Coverage for collections (i.e. programs in collection/) does not seem to work at
       the moment. Therefore, they often end up with (next to) zero coverage in the generated
       reports.

       Caveat 2: Devel::Cover sometimes changes the output of Lintian or tools called by Lintian.
       Obviously, this leads to test failures. Therefore, you may see weird test failures (or
       warnings) when running under coverage.

       Collecting the coverage you want in a reasonable time

       Collecting coverage is excruciatingly slow.  This is not helped by the fact that it
       becomes unreliable when run under 2 or more threads.

       Fortunately, Devel::Cover "appends" to its cover database.  This allows you to "slowly"
       build up the coverage database over multiple runs. Example:

         $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:scripts
         $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:debs
         $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:source
         ...

       Or:

         $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out $check
         $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out legacy

SEE ALSO

       Lintian::Tutorial::WritingTests