Provided by: cmdtest_0.32-1_all bug


       cmdtest - blackbox testing of Unix command line tools


       cmdtest      [-cCOMMAND]      [--command=COMMAND]      [--config=FILE]     [--dump-config]
       [--dump-setting-names]   [--generate-manpage=TEMPLATE]    [-h]    [--help]    [--help-all]
       [--list-config-files]  [--version]  [--no-default-configs]  [--dump-memory-profile=METHOD]
       [-k] [--keep] [--no-keep] [--log=FILE] [--log-keep=N] [--log-level=LEVEL] [--log-max=SIZE]
       [--log-mode=MODE]  [--memory-dump-interval=SECONDS] [--output=FILE] [-tTEST] [--test=TEST]
       [--timings] [--no-timings] [FILE]...


       cmdtest black box tests Unix command line tools.  Given some test scripts,  their  inputs,
       and  expected outputs, it verifies that the command line produces the expected output.  If
       not, it reports problems, and shows the differences.

       Each test case foo consists of the following files:

              a script to run the test (this is required)

              the file fed to standard input

              the expected output to the standard output

              the expected output to the standard error

              the expected exit code

              a shell script to run before the test

              a shell script to run after test

       Usually, a single test is not enough. All tests are put into the same directory, and  they
       may share some setup and teardown code:

              a shell script to run once, before any tests

       setup  a shell script to run before each test

              a shell script to run after each test

              a shell script to run once, after all tests

       cmdtest  is  given  the  name  of  the  directory  with  all  the  tests,  or several such
       directories, and it does the following:

       • execute setup-once

       • for each test case (unique prefix foo):

              — execute setup

              — execute foo.setup

              — execute the command, by running foo.script, and redirecting standard input to
                come from foo.stdin, and capturing standard output and error and exit codes

              — execute foo.teardown

              — execute teardown

              — report result of test: does exit code match foo.exit, standard output match
                foo.stdout, and standard error match foo.stderr?

       • execute teardown-once

       Except for foo.script, all of these files are optional.  If a setup or teardown script is
       missing, it is simply not executed.  If one of the standard input, output, or error files
       is missing, it is treated as if it were empty.  If the exit code file is missing, it is
       treated as if it specified an exit code of zero.

       The shell scripts may use the following environment variables:

              a temporary directory where files may be created by the test

              name of the current test (will be empty for setup-once and teardown-once)

       SRCDIR directory from which cmdtest was launched


       -c, --command=COMMAND
              ignored for backwards compatibility

              fill in manual page TEMPLATE

       -h, --help
              show this help message and exit

       -k, --keep
              keep temporary data on failure

              opposite of --keep

              write output to FILE, instead of standard output

       -t, --test=TEST
              run only TEST (can be given many times)

              report how long each test takes

              opposite of --timings

              show program's version number and exit

   Configuration files and settings
              add FILE to config files

              write out the entire current configuration

              write out all names of settings and quit

              show all options

              list all possible config files

              clear list of configuration files to read

              write log entries to FILE (default is to not write log files at all); use "syslog"
              to log to system log, "stderr" to log to the standard error output, or "none" to
              disable logging

              keep last N logs (10)

              log at LEVEL, one of debug, info, warning, error, critical, fatal (default: debug)

              rotate logs larger than SIZE, zero for never (default: 0)

              set permissions of new log files to MODE (octal; default 0600)

              make memory profiling dumps using METHOD, which is one of: none, or simple (no
              meliae support anymore)(default: simple)

              make memory profiling dumps at least SECONDS apart


       To test that the echo(1) command outputs the expected string, create a file called echo-
       tests/hello.script containing the following content:

              echo hello, world

       Also create the file echo-tests/hello.stdout containing:

              hello, world

       Then you can run the tests:

              $ cmdtest echo-tests
              test 1/1
              1/1 tests OK, 0 failures

       If you change the stdout file to be something else, cmdtest will report the differences:

              $ cmdtest echo-tests
              FAIL: hello: stdout diff:
              --- echo-tests/hello.stdout   2011-09-11 19:14:47 +0100
              +++ echo-tests/hello.stdout-actual 2011-09-11 19:14:49 +0100
              @@ -1 +1 @@
              -something else
              +hello, world

              test 1/1
              0/1 tests OK, 1 failures

       Furthermore, the echo-tests directory will contain the actual output files, and diffs from
       the expected files.  If one of the actual output files is actually correct, you can
       actualy rename it to be the expected file.  Actually, that's a very convenient way of
       creating the expected output files: you run the test, fixing things, until you've manually
       checked the actual output is correct, then you rename the file.