Provided by: aegis_4.24.3-3_amd64 bug

NAME

        aegis test - run tests

SYNOPSIS

        aegis -Test [ option...  ][ name=value ][ file-name...  ]
        aegis -Test -INDependent [ option...  ][ name=value ][ file-name...  ]
        aegis -Test -List [ option...  ]
        aegis -Test -Help

DESCRIPTION

        The aegis -Test command is used to run tests.  If no files are named, all relevant tests
        are run.  By default both automatic and manual tests are run.

        You may name directories on the command line, and all relevant tests in that directory
        tree in the change will be run.  It is an error if there are no relevant tests.

        Each architecture must be tested separately.  This is because there may be subtle
        problems that are only revealed on some architectures.  Some projects may also have
        different code for different architectures.

        The status of the last test run is remembered so that tests are not run if there is no
        need.  (This does not apply to -REGression tests, unfortunately.)  Tests must be re-run
        if the test previously failed, if the test file has changed, if there has been a build,
        and for each architecture.

   name=value
        You can add name=value pairs to the command line, these will be passed unchanged to the
        test command.  Usually on the end of the command line, but this can be changed in the
        project configuration file.

        The -force option results in an implicit force=1 variable being added to the list of
        variable assignments, and thus added to the end of the command.  This is of most use when
        using the batch_test_command filed of the project configuration file.

        This may initially look like a development process end-run, allowing test scripts to be
        written so that they give all the right answers without actually doing anything.  You
        have always been able to do this with environment variables, so this isn't anything new.

        It is possible to get all of the variable assignments to turn into environment variables
        by putting $var at the start of the command, before the name of the shell, rather than at
        the default location at the end of the command.

   File Name Interpretation
        The aegis program will attempt to determine the project file names from the file names
        given on the command line.  All file names are stored within aegis projects as relative
        to the root of the baseline directory tree.  The development directory and the
        integration directory are shadows of this baseline directory, and so these relative names
        apply here, too.  Files named on the command line are first converted to absolute paths
        if necessary.  They are then compared with the baseline path, the development directory
        path, and the integration directory path, to determine a baseline-relative name.  It is
        an error if the file named is outside one of these directory trees.

        The -BAse_RElative option may be used to cause relative filenames to be interpreted as
        relative to the baseline path; absolute filenames will still be compared with the various
        paths in order to determine a baseline-relative name.

        The relative_filename_preference in the user configuration file may be used to modify
        this default behavior.  See aeuconf(5) for more information.

TEST PROCESS

        Each change is required to be accompanied by tests, and those tests are required to be
        run against the built development directory, and they must pass.  This ensures that new
        functionality is accompanied by tests to verify its correctness, and bug fixes are
        accompanied by tests which confirm that the bug has been fixed.

   Regression Tests
        Tests are treated as any other source file, and are maintained in the baseline and
        history with all other source files.  The tests which must accompany every change
        accumulate in the project baseline, providing a definition of correct function for the
        baseline.  These accumulated tests may be executed using an “aegis -REGression” command,
        to verify that the project will not “regress” as a result of a change.

   Baseline Tests
        Bug fixes are required to have their tests fail against the project baseline (in contrast
        to the development directory).  This ensures that the test actually demonstrates the bug
        in the baseline, as well as demonstrating that it is fixed by the change.  New
        functionality trivially fails against the baseline, and so aegis does not attempt to
        guess if a test is a bug fix test or new functionality test, it simply requires tests to
        fail against the baseline.

        This requirement applies both to new tests being created by a change and also to tests
        which have been copied into a change for modification.

   Reviewing Tests
        Reviewers may be confident that aegis has enforced the test requirements; that a change
        must have tests, that the change must build, that the tests pass against the development
        directory, and that the tests fail against the baseline.  These conditions are enforced
        by aede(1) and the change will not be advanced to the being reviewed state until these
        conditions are met.  Reviewers should thus review tests for completeness of coverage of
        the code in the change, and insensitivity to changes in the execution environment (e.g.
        not date sensitive).  Reviewers should also use “aegis -list change_details” to verify
        that a change does or does not have testing exemptions.

   Exemptions
        Various test exemptions may be granted by project administrators, see aepa(1) and
        aepattr(5) for more information.  Copying tests into a change, or adding new tests to a
        change, may cancel those exemptions.

TEST COMMAND CONFIGURATION

        The command used to execute tests is defined by the test_command field in the project
        configuration file (see aepconf(5) for more information), this defaults to using the
        Bourne shell if not set.  The current directory will be the top of the appropriate
        directory tree.  If tests require temporary files, they should create them in /tmp, as a
        test cannot expect to have write permission in the current directory.

        If you want to use a more sophisticated test engine, rather than a simple shell script,
        but this test engine does not return result codes suitable for use with aegis, you could
        wrap it in a shell script which re-writes the exit status into the values aegis expects.
        You could also achieve the same results by writing a more complex test_command in the
        project config file.

        It is also possible to write test commands which are able to test more than one file at
        once.  This is controlled by the batch_test_command field of the project config file.  In
        this case, the ${output} substitution indicates the name of a file the test command must
        create, in aetest(5) format, to contain the results of the tests run.  This is often used
        on systems with multiple CPUs or the ability to distribute jobs across several computers
        on a network.

   Substitutions
        All of the aesub(5) substitutions are available in the test commands.  Some of them are
        of particular note:

        ARCHitecture
                This substitution is replaced by the name of the architecture to be tested.

        Search_Path
                This substitution is replaced by a colon separated list of absolute paths to
                search when looking for test support files.

        Search_Path_Executable
                This substitution is replaced by a colon separated list of absolute paths to
                search when looking for executable support files (library files and sub-
                commands).

        Most of the time $Search_Path_Executable are exactly the same.  However, during “aegis -t
        -bl” they will be different, with $Seach_Path starting at the development directory (the
        test being run) and $Seach_Path_Executable starting at the baseline (the executable being
        run).

   Test Result Codes
        As each test is run (via the test_command field in the project config file), aegis
        determines whether the test succeeded or failed by looking at its exit status.  This exit
        status is mostly as expected for UNIX commands.

        Success
            A test should exit 0 to indicate success, i.e. that the specific function under test
            worked as expected.

        Failure
            A test should exit 1 to indicate failure, i.e. that the specific function under test
            did not work as expected.

        No Result
            A test should exit 2 to indicate no result, i.e. that the specific function under
            test could not be exercised because something else went wrong.  For example, running
            out of disk space when creating the test input files in the /tmp directory.

        Skipped
            A test should exit 77 to indicate that it was skipped.  This is usually to do with
            the current architecture not being meaningful.  Whenever possible, use “No Result”
            instead.  (The value was chosen for compatibility with other test systems.)

        Actually, any exit code other than 0, 1 or 77 will be interpreted as “no result”.
        However, always using 0, 1, 2 or 77 means that if a new result code is required by a
        later release of Aegis your existing tests will continue to work.

TEST CORRELATIONS

        The “aegis -Test -SUGgest” command may be used to have aegis suggest suitable regression
        tests for your change, based on the source files in your change.  This automatically
        focuses testing effort to relevant tests, reducing the number of regression tests
        necessary to be confident that you have not introduced a bug.

        The test correlations are generated by the “aegis -Integrate_Pass” command, which
        associates each test in the change with each source file in the change.  Thus, each
        source file accumulates a list of tests which have been associated with it in the past.
        This is not as exact as code coverage analysis, but is a reasonable approximation in
        practice.

        The aecp(1) and aenf(1) commands are used to associate files with a change.  While they
        do not actively perform the association, these are the files used by aeipass(1) and
        aet(1) to determine which source files are associated with which tests.

   Test Correlation Accuracy
        Assuming that the testing correlations are accurate and that the tests are evenly
        distributed across the function space, there will be a less than 1/number chance that a
        relevant test has not been run by the “aegis -Test -SUGgest number” command.  A small
        amount of noise is added to the test weighting, so that unexpected things are sometimes
        tested, and the same tests are not run every time.

        Test correlation accuracy can be improved by ensuring that:

        • Each change should be strongly focused, with no gratuitous file inclusions.  This
          avoids spurious correlations.

        • Each item of new functionality should be added in an individual change, rather than
          several together.  This strongly correlates tests with functionality.

        • Each bug should be fixed in an individual change, rather than several together.  This
          strongly correlates tests with functionality.

        • Test correlations will be lost if files are moved.  This is because correlations are by
          name.

        The best way for tests to correlate accurately with source files is when a change
        contains a test and exactly those files relating to the functionality under test.  Too
        many spurious files will weaken the usefulness of the testing correlations.

OPTIONS

        The following options are understood:

        -AUTOmatic
                This option may be used to specify automatic tests.  Automatic tests require no
                human assistance.

        -BaseLine
                This option may be used to specify that the project baseline is the subject of
                the command.

        -BAse_RElative
                This option may be used to cause relative filenames to be considered relative to
                the base of the source tree.  See aeuconf(5) for the corresponding user
                preference.

        -CUrrent_RElative
                This option may be used to cause relative filenames to be considered relative to
                the current directory.  This is usually the default.  See aeuconf(5) for the
                corresponding user preference.

        -Change number
                This option may be used to specify a particular change within a project.  See
                aegis(1) for a complete description of this option.

        -FOrce  This option may be used to specify that all tests should be run, even if the
                status of the last test run indicates that there is no need to run a specific
                test.

        -Help
                This option may be used to obtain more information about how to use the aegis
                program.

        -INDependent
                This option is used to specify that the test is to be run independent of any
                particular change.  If no tests are named, all tests in the baseline will be run.

        -List
                This option may be used to obtain a list of suitable subjects for this command.
                The list may be more general than expected.

        -MANual This option may be used to specify manual tests.  Manual tests require some human
                intervention,  e.g.: confirmation of some screen behavior (X11, for instance), or
                some user action, "unplug ethernet cable now".

        -Not_Logging
                This option may be used to disable the automatic logging of output and errors to
                a file.  This is often useful when several aegis commands are combined in a shell
                script.

        -PErsevere
                This option may be used to specify that all tests should be run, even if some
                fail.  Defaults to the user's persevere_preference if not specified, see
                aeuconf(5) for more information.

        -No_PErsevere
                This option may be used to specify that the test run should stop after the first
                failure.  Defaults to the user's persevere_preference if not specified, see
                aeuconf(5) for more information.

        -Project name
                This option may be used to select the project of interest.  When no -Project
                option is specified, the AEGIS_PROJECT environment variable is consulted.  If
                that does not exist, the user's $HOME/.aegisrc file is examined for a default
                project field (see aeuconf(5) for more information).  If that does not exist,
                when the user is only working on changes within a single project, the project
                name defaults to that project.  Otherwise, it is an error.

        -PROGress
                This option may be used to specify that progress messages should be issued before
                each test run or before each batch test run in case batch_test_command field
                specified in project config file (see aeuconf(5) for more information).

        -No_PROGress
                This option may be used to specify that progress messages should be suppressed.
                This is the default.

        -REGression
                This option is used to specify that the regression test suite is to be run.  The
                regression test suite consists of all tests in the baseline which do not appear
                in the change.  It is an error if there are no regression tests.  You may not
                name tests on the command line when using the -REGression option.  You may name
                individual tests to be run on the command line, without using the -REGression
                option; if they are not part of the change, the tests of the same name in the
                baseline will be run.

        -SUGgest [ number ]
                The “aegis -Integrate_Pass” command collects test correlation statistics when
                changes are integrated.  This option may be used to request that aegis suggest
                which tests should be run, using these testing correlations.  If no number is
                specified, 10 tests will be suggested.  This option implies the -REGression
                option.

        -SUGgest_Limit minutes
                This option may be used to limit the number of tests to a certain number of
                minutes.  They will be run from most relevant to least relevant.

        -SUGgest_Noise number
                This option may be used to control the amount of noise injected into the test
                selection performed by the -SUGgest option.  The number is a percentage of noise
                to be injected.  Defaults to 10 if not specified.  The injection of noise ensures
                that a variety of tests are run on subsequent runs, and also some from left-field
                as a sanity check.

        -TERse
                This option may be used to cause listings to produce the bare minimum of
                information.  It is usually useful for shell scripts.

        -Verbose
                This option may be used to cause aegis to produce more output.  By default aegis
                only produces output on errors.  When used with the -List option this option
                causes column headings to be added.

        -Wait   This option may be used to require Aegis commands to wait for access locks, if
                they cannot be obtained immediately.  Defaults to the user's lock_wait_preference
                if not specified, see aeuconf(5) for more information.

        -No_Wait
                This option may be used to require Aegis commands to emit a fatal error if access
                locks cannot be obtained immediately.  Defaults to the user's
                lock_wait_preference if not specified, see aeuconf(5) for more information.

        See also aegis(1) for options common to all aegis commands.

        All options may be abbreviated; the abbreviation is documented as the upper case letters,
        all lower case letters and underscores (_) are optional.  You must use consecutive
        sequences of optional letters.

        All options are case insensitive, you may type them in upper case or lower case or a
        combination of both, case is not important.

        For example: the arguments "-project, "-PROJ" and "-p" are all interpreted to mean the
        -Project option.  The argument "-prj" will not be understood, because consecutive
        optional characters were not supplied.

        Options and other command line arguments may be mixed arbitrarily on the command line,
        after the function selectors.

        The GNU long option names are understood.  Since all option names for aegis are long,
        this means ignoring the extra leading '-'.  The "--option=value" convention is also
        understood.

RECOMMENDED ALIAS

        The recommended alias for this command is
        csh%    alias aet 'aegis -t \!* -v'
        sh$     aet(){aegis -t "$@" -v}

ERRORS

        It is an error if the change is not in one of the being developed or being integrated
        states.
        It is an error if the change is not assigned to the current user.
        It is an error if your have no relevant tests and no relevant exemption.

EXIT STATUS

        The aegis command will exit with a status of 1 on any error.  The aegis command will only
        exit with a status of 0 if there are no errors.

ENVIRONMENT VARIABLES

        See aegis(1) for a list of environment variables which may affect this command.  See
        aepconf(5) for the project configuration file's project_specific field for how to set
        environment variables for all commands executed by Aegis.

SEE ALSO

        aeb(1)  build a change

        aeca(1) modify the attributes of a change

        aedb(1) begin development of a change

        aeib(1) begin integration of a change

        aent(1) add a new test to a change

        aecp(1) copy an existing test into a change

        aepconf(5)
                project configuration file format

        aeuconf(5)
                user configuration file format

COPYRIGHT

        aegis version 4.24.3.D001
        Copyright (C) 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002,
        2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 Peter Miller

        The aegis program comes with ABSOLUTELY NO WARRANTY; for details use the 'aegis -VERSion
        License' command.  This is free software and you are welcome to redistribute it under
        certain conditions; for details use the 'aegis -VERSion License' command.

AUTHOR

        Peter Miller   E-Mail:   millerp@canb.auug.org.au
        /\/\*             WWW:   http://www.canb.auug.org.au/~millerp/