Provided by: autopkgtest_5.32ubuntu3~23.10.1_all 
NAME
autopkgtest - test an installed binary package using the source package's tests
SYNOPSIS
autopkgtest [options...] [testbinary...] testsrc -- virt-server [virt-server-arg...]
DESCRIPTION
autopkgtest runs tests on binary Debian packages, as installed on a system (called
"testbed"). The tests are supplied in the source package.
autopkgtest runs each test supplied by a particular package and reports the results. It
drives the specified virtualisation regime as appropriate, parses the test description
metadata, and arranges for data to be copied to and from the testbed as required.
See /usr/share/doc/autopkgtest/README.running-tests.rst.gz for an introduction about how
to use autopkgtest.
TESTING A DEBIAN PACKAGE
Positional (non-option) arguments specify exactly one source package (containing the test
code) and optionally some binary packages to test.
testsrc can be one of:
.dsc file
Run tests from Debian .dsc source package. By default the package will also be
built and the resulting binaries will be used to satisfy test dependencies; to
disable that, specify the -B/--no-built-binaries option.
source package directory
Run tests from a Debian source tree directory. If that is an unbuilt tree, this is
very similar to specifying a .dsc. If that is a built tree, all test dependencies
get satisfied by archive packages, unless you explicitly specify locally built
.debs as well.
Attention: If you just specify a bare directory name which is a legal Debian source
package name, it will be interpreted as the latter (see below). In this case,
prefix the directory name with ./.
current directory
If no source package is specified on the command line and the current directory is
a Debian source package, this will be tested.
source package name
Downloads the given source package name with apt-get source in the testbed and run
its tests. This is similar to specifying a .dsc but avoids copying the source from
the host to the testbed. Possibly built binaries (if the test specifies
build-needed) will not be used to satisfy dependencies, as usually in this mode you
want to test binaries from a real archive.
git URL or URL#branch
Git-clones the given URL (which must contain an unbuilt Debian source tree) and
runs the tests from that. If branch is given, this branch will be checked out
instead of the default (usually "master"). This can also be a more general refspec
such as #refs/pull/123/head" for a GitHub pull request.
This is very similar to cloning manually and specifying the checkout directory as
test; i. e. this is commonly used with --no-built-binaries. The git package will
be installed if necessary.
.changes file
Run tests from the .dsc source package in the given .changes file. If the .changes
contains .deb packages, they will be used for the test. Acts as if you had
specified the .debs and .dsc from the .changes file as explicit arguments. Note
that if the .changes contains only debs, the corresponding .dsc still needs to be
specified alongside, or the current directory must be the source package.
All other positional arguments must be .deb binary packages. They will be used for both
build and test dependencies of the source package. If any binary package is given, then
--no-built-binaries is implied.
TEST OPTIONS
-a ARCH | --architecture=ARCH
Make the testbed a multiarch system, setup the environment for cross-building
(possibly installing extra build dependencies), and run tests for the specified
foreign architecture rather than for the testbed native architecture.
Attention: Specifying this option does not forward anything to the virt-server, so
you may want to look at your virt-server's options to give it the architecture too.
-B | --no-built-binaries
Binaries from unbuilt source packages (see above) will not be built or ignored, and
dependencies are satisfied with packages from the archive. Note that the source
package still gets built if a test requires build-needed.
--override-control=PATH
Read the test metadata from PATH instead of debian/tests/control.
--test-name=TEST
Run only the given test name (from test control file). If this option is used more
than once, all the named tests are run. This replaces --testname, which is
deprecated.
--skip-test=TEST
Skip the given test name (from test control file). If this option is used more
than once, all the named tests are skipped.
LOGGING OPTIONS
If you don't specify any option, autopkgtest only writes its output/results to stderr.
-o dir | --output-dir=dir
Specifies that test artifacts (stderr and stdout from the tests, the log file,
built binary packages etc.) should be placed in the given directory. dir must not
exist yet or be empty, otherwise autopkgtest will refuse to use it.
-l logfile | --log-file=logfile
Specifies that the trace log should be written to logfile instead of to output-dir.
--summary=summary
Specifies that a summary of the outcome should be written to summary. The events
in the summary are written to the log in any case.
-q | --quiet
Do not send a copy of autopkgtest's trace logstream to stderr. This option does
not affect the copy sent to logfile or output-dir. Note that without the trace
logstream it can be very hard to diagnose problems.
TEST BED SETUP OPTIONS
--setup-commands=commands
Run commands after opening the testbed. This can be used to do anything that isn't
supported by an existing autopkgtest command. If commands is an existing file name,
the commands are read from that; otherwise it is a string with the actual commands
that gets run as-is. File names without directory will be searched in both the
current directory and in /usr/share/autopkgtest/setup-commands/ so you do not need
to give the full path for setup scripts shipped with autopkgtest.
Normally, if the setup commands fail, autopkgtest will consider this a transient
testbed error (exit code 16). However, if the setup commands exit with code 100,
autopkgtest will consider this an "erroneous package" (exit code 12) instead, so
this can be used to e. g. detect upgrade errors to a new version. Note that apt
exits with exit code 100 in these cases.
This option can be specified multiple times.
If --user is given or the test bed provides a suggested-normal-user capability, the
$AUTOPKGTEST_NORMAL_USER environment variable will be set to that user.
If the setup commands affect anything in boot directories (like /boot or
/lib/systemd/system) and the testbed supports rebooting, the testbed will be
rebooted after the setup commands. This can be suppressed by creating a file
/run/autopkgtest_no_reboot.stamp.
--setup-commands-boot=commands
Run commands after the --setup-commands, and after every reboot. For example, these
commands could be used to add files in a tmpfs.
These commands never cause the testbed to be rebooted (because that could lead to
an infinite loop). Otherwise, they are just like the --setup-commands.
This option can be specified multiple times.
--add-apt-source='deb http://MIRROR SUITE COMPONENT...'
Add the given apt source to /etc/apt/sources.list.d and update it, before running
any --setup-commands.
This option can be specified multiple times.
--add-apt-release='RELEASE'
Add the given apt RELEASE to /etc/apt/sources.list.d and update it, before running
any --setup-commands. The mirror and components to use are copied from the very
first existing APT sources.list entry. Both binary ("deb") and source ("deb-src")
entries are added.
This option can be specified multiple times.
--apt-upgrade | -U
Run apt-get update and apt-get dist-upgrade -y in the testbed before running the
tests. Any --add-apt-source or --apt-pocket options take effect first, so this
will upgrade packages from those sources if appropriate.
--apt-default-release=SUITE
Set's APT::Default-Release value to the provided value. For apt pinning (related to
--apt-pocket, and --pin-packages) to work properly, APT::Default-Release must be
set to the release that should provide the packages that are not pinned. For Debian
and Ubuntu, this is normally automatically detected from the first entry in
/etc/apt/sources.list.
--apt-pocket=pocket[=pkgname,src:srcname,...]
Add apt sources for release-pocket. This finds the first deb line in
/etc/apt/sources.list which does not already specify a pocket and adds a deb and
deb-src line with that pocket to /etc/apt/sources.list.d/pocket.list. This also
calls apt-get update for the new pocket (but not for anything else). The pocket
will be pinned with Pin-Priority: 500, so the "NotAutomatic: yes" setting will have
no effect on the testbed system.
If a package list is given after =, set up apt pinning to use only those packages
from pocket. An entry "src:srcname" expands to all binary packages built by that
source. This can be used for minimizing dependencies taken from pocket so that
package updates in that pocket can be tested independently from each other for
better isolation. Attention: This does not currently resolve some situations where
dependencies of the given packages can only be resolved in the given pocket. In
that case the apt pinning will be removed and package installation will be retried
with the entirety of pocket.
--copy=HOSTPATH:TESTBEDPATH
Copy file or directory from host into testbed after opening. This happens before
--setup-commands thus you can use these files in the setup commands.
--env=VAR=value
Set arbitrary environment variable in the build and test. Can be specified multiple
times.
--pin-packages=RELEASE=PACKAGE[,PACKAGE2]
Set up apt pinning to use only those packages from RELEASE. An entry "src:srcname"
expands to all binary packages built by that source. This can be used for
minimizing dependencies taken from RELEASE so that package updates in that release
can be tested independently from each other for better isolation.
--no-apt-fallback
Disable the apt-get fallback which is used with --apt-pocket or --pin-packages in
case installation of dependencies fails due to strict pinning.
--ignore-restrictions=RESTRICTION,RESTRICTION...
If a test would normally be skipped because it has Restrictions: RESTRICTION, run
it anyway. Can be specified multiple times.
For example, you might ignore the restriction isolation-machine when using the null
virtualization server if you know that autopkgtest itself is running on an
expendable virtual machine. These options also work for unknown restrictions, so
they can be used when experimenting with new restrictions.
USER/PRIVILEGE HANDLING OPTIONS
-u user | --user=user
Run builds and tests as user on the testbed. This needs root on the testbed; if
root on the testbed is not available then builds and tests run as whatever user is
provided.
--gain-root=gain-root
Prefixes debian/rules binary with gain-root. The default is not to use anything,
except that if --user is supplied or root on the testbed is not available the
default is fakeroot.
DEBUGGING OPTIONS
--debug|-d
Include additional debugging information in the trace log. Each additional -d
increases the debugging level; the current maximum is -ddd. If you like to see
what's going on, -d or -dd is recommended.
--shell-fail|-s
Run an interactive shell in the testbed after a failed build, test, or dependency
installation.
--shell
Run an interactive shell in the testbed after every test.
TIMEOUT OPTIONS
--timeout=seconds
Define the global timeout, limiting the time allowed to perform the build (if one
is required) and run the tests. If the global timeout is reached during the build,
the run will be aborted in the same way as if the build had failed. If the global
timeout is exceeded while running a test, the test is aborted and any remaining
tests are skipped. (default: 0s, meaning a global timeout is not in effect). The
value must be specified as an integer number of seconds.
--timeout-which=seconds
Use a different timeout for operations on or with the testbed. There are five
timeouts affected by five values of which: short: supposedly short operations like
setting up the testbed's apt and checking the state (default: 100s); install:
installation of packages including dependencies (default: 3,000s); test: test runs
(default: 10,000s); copy: copy files/directories between host and testbed (default:
300s); and build: builds (default: 100,000s). The value must be specified as an
integer number of seconds.
--timeout-factor=double
Multiply all of the default timeouts by the specified factor (see --timeout-which
above). Only the defaults are affected; explicit timeout settings are used exactly
as specified.
LOCALE OPTIONS
--set-lang=langval
When running commands on the testbed, sets the LANG environment variable to
langval. The default in autopkgtest is to set it to C.UTF-8.
OTHER OPTIONS
--no-auto-control
Disable automatic test generation with autodep8, even if it is installed. In that
case, packages without tests will exit with code 8 ("No tests in this package")
just like without autodep8.
--build-parallel=N
Set parallel=N DEB_BUILD_OPTION for building packages. By default this is the
number of available processors. This is mostly useful in containers where you can
restrict the available RAM, but not restrict the number of CPUs.
--needs-internet=run|try|skip
Define how to handle the needs-internet restriction. With "try" tests with
needs-internet restrictions will be run, but if they fail they will be treated as
flaky tests. With "skip" these tests will skipped immediately and will not be run.
With "run" the restriction is basically ignored, this is the default.
-V|--validate
Validate the test control file and exit without running any tests.
-h|--help
Show command line help and exit.
VIRTUALIZATION SERVER
-- virt-server virt-server-arg...
Specifies the virtualisation regime server, as a command and arguments to invoke.
virt-server must be an existing autopkgtest virtualization server such as schroot
or qemu.
All the remaining arguments and options after -- are passed to the virtualisation
server program. See the manpages of the individual servers for how to use them.
OUTPUT FORMAT
During a normal test run, one line is printed for each test. This consists of a short
string identifying the test, some horizontal whitespace, and one of PASS, PASS details,
FAIL reason, SKIP reason, or FLAKY reason where the pass/fail indication is separated by
any reason or details by some horizontal whitespace.
The string to identify the test consists of a short alphanumeric string invented by
autopkgtest to distinguish different command-line arguments, the argid, followed by a
hyphen and the test name.
SKIP indicates that a test was not run, or that the test code was started but detected
that the test could not complete, for instance because a required resource was not
available.
FLAKY indicates that a test would ordinarily have failed, but because this particular test
is known to be unreliable, the failure was ignored.
Sometimes a SKIP will be reported when the name of the test is not known or not
applicable: for example, when there are no tests in the package, or a there is a test
stanza which contains features not understood by this version of autopkgtest. In this
case * will appear where the name of the test should be.
If autopkgtest detects that erroneous package(s) are involved, it will print the two lines
blame: blamed-thing... and badpkg: message. Here each whitespace-separated blamed-thing
is one of arg:argument (representing a pathname found in a command line argument),
dsc:package (a source package name), deb:package (a binary package name) or possibly other
strings to be determined. This indicates which arguments and/or packages might have
contributed to the problem; the ones which were processed most recently and which are
therefore most likely to be the cause of a problem are listed last.
CONFIGURATION FILES
If you use lots of options or nontrivial virt server arguments, you can put any part of
the command line into a text file, with one line per option. E. g. you can create a file
sid.cfg with contents like
-s
--output-dir=/tmp/testout
--apt-upgrade
--
schroot
sid
and then run
autopkgtest foo_1_amd64.changes @sid.cfg
The contents of the configuration file will be expanded in-place as if you would have
given its contents on the command line. Please ensure that you don't place spaces between
short options and their values, they would become a part of the argument value.
EXIT STATUS
0 all tests passed
2 at least one test was skipped (or at least one flaky test failed)
4 at least one test failed
6 at least one test failed and at least one test skipped
8 no tests in this package, or all non-superficial tests were skipped
12 erroneous package
14 erroneous package and at least one test skipped
16 testbed failure
20 other unexpected failures including bad usage
SEE ALSO
/usr/share/doc/autopkgtest/README.running-tests.rst.gz
/usr/share/doc/autopkgtest/README.package-tests.rst.gz
AUTHORS AND COPYRIGHT
This manpage is part of autopkgtest, a tool for testing Debian binary packages.
autopkgtest is Copyright (C) 2006-2014 Canonical Ltd.
See /usr/share/doc/autopkgtest/CREDITS for the list of contributors and full copying
conditions.