Provided by: aegis_4.24.3-3_i386 bug

NAME

        aegis test - run tests

SYNOPSIS

        aegis -Test [ option...  ][ name=value ][ file-name...  ]
        aegis -Test -INDependent [ option...  ][ name=value ][ file-name...  ]
        aegis -Test -List [ option...  ]
        aegis -Test -Help

DESCRIPTION

        The aegis -Test command is used to run tests.  If no files are named,
        all relevant tests are run.  By default both automatic and manual
        tests are run.

        You may name directories on the command line, and all relevant tests
        in that directory tree in the change will be run.  It is an error if
        there are no relevant tests.

        Each architecture must be tested separately.  This is because there
        may be subtle problems that are only revealed on some architectures.
        Some projects may also have different code for different
        architectures.

        The status of the last test run is remembered so that tests are not
        run if there is no need.  (This does not apply to -REGression tests,
        unfortunately.)  Tests must be re-run if the test previously failed,
        if the test file has changed, if there has been a build, and for each
        architecture.

   name=value
        You can add name=value pairs to the command line, these will be passed
        unchanged to the test command.  Usually on the end of the command
        line, but this can be changed in the project configuration file.

        The -force option results in an implicit force=1 variable being added
        to the list of variable assignments, and thus added to the end of the
        command.  This is of most use when using the batch_test_command filed
        of the project configuration file.

        This may initially look like a development process end-run, allowing
        test scripts to be written so that they give all the right answers
        without actually doing anything.  You have always been able to do this
        with environment variables, so this isn't anything new.

        It is possible to get all of the variable assignments to turn into
        environment variables by putting $var at the start of the command,
        before the name of the shell, rather than at the default location at
        the end of the command.

   File Name Interpretation
        The aegis program will attempt to determine the project file names
        from the file names given on the command line.  All file names are
        stored within aegis projects as relative to the root of the baseline
        directory tree.  The development directory and the integration
        directory are shadows of this baseline directory, and so these
        relative names apply here, too.  Files named on the command line are
        first converted to absolute paths if necessary.  They are then
        compared with the baseline path, the development directory path, and
        the integration directory path, to determine a baseline-relative name.
        It is an error if the file named is outside one of these directory
        trees.

        The -BAse_RElative option may be used to cause relative filenames to
        be interpreted as relative to the baseline path; absolute filenames
        will still be compared with the various paths in order to determine a
        baseline-relative name.

        The relative_filename_preference in the user configuration file may be
        used to modify this default behavior.  See aeuconf(5) for more
        information.

TEST PROCESS

        Each change is required to be accompanied by tests, and those tests
        are required to be run against the built development directory, and
        they must pass.  This ensures that new functionality is accompanied by
        tests to verify its correctness, and bug fixes are accompanied by
        tests which confirm that the bug has been fixed.

   Regression Tests
        Tests are treated as any other source file, and are maintained in the
        baseline and history with all other source files.  The tests which
        must accompany every change accumulate in the project baseline,
        providing a definition of correct function for the baseline.  These
        accumulated tests may be executed using an "aegis -REGression"
        command, to verify that the project will not "regress" as a result of
        a change.

   Baseline Tests
        Bug fixes are required to have their tests fail against the project
        baseline (in contrast to the development directory).  This ensures
        that the test actually demonstrates the bug in the baseline, as well
        as demonstrating that it is fixed by the change.  New functionality
        trivially fails against the baseline, and so aegis does not attempt to
        guess if a test is a bug fix test or new functionality test, it simply
        requires tests to fail against the baseline.

        This requirement applies both to new tests being created by a change
        and also to tests which have been copied into a change for
        modification.

   Reviewing Tests
        Reviewers may be confident that aegis has enforced the test
        requirements; that a change must have tests, that the change must
        build, that the tests pass against the development directory, and that
        the tests fail against the baseline.  These conditions are enforced by
        aede(1) and the change will not be advanced to the being reviewed
        state until these conditions are met.  Reviewers should thus review
        tests for completeness of coverage of the code in the change, and
        insensitivity to changes in the execution environment (e.g. not date
        sensitive).  Reviewers should also use "aegis -list change_details" to
        verify that a change does or does not have testing exemptions.

   Exemptions
        Various test exemptions may be granted by project administrators, see
        aepa(1) and aepattr(5) for more information.  Copying tests into a
        change, or adding new tests to a change, may cancel those exemptions.

TEST COMMAND CONFIGURATION

        The command used to execute tests is defined by the test_command field
        in the project configuration file (see aepconf(5) for more
        information), this defaults to using the Bourne shell if not set.  The
        current directory will be the top of the appropriate directory tree.
        If tests require temporary files, they should create them in /tmp, as
        a test cannot expect to have write permission in the current
        directory.

        If you want to use a more sophisticated test engine, rather than a
        simple shell script, but this test engine does not return result codes
        suitable for use with aegis, you could wrap it in a shell script which
        re-writes the exit status into the values aegis expects.  You could
        also achieve the same results by writing a more complex test_command
        in the project config file.

        It is also possible to write test commands which are able to test more
        than one file at once.  This is controlled by the batch_test_command
        field of the project config file.  In this case, the ${output}
        substitution indicates the name of a file the test command must
        create, in aetest(5) format, to contain the results of the tests run.
        This is often used on systems with multiple CPUs or the ability to
        distribute jobs across several computers on a network.

   Substitutions
        All of the aesub(5) substitutions are available in the test commands.
        Some of them are of particular note:

        ARCHitecture
                This substitution is replaced by the name of the architecture
                to be tested.

        Search_Path
                This substitution is replaced by a colon separated list of
                absolute paths to search when looking for test support files.

        Search_Path_Executable
                This substitution is replaced by a colon separated list of
                absolute paths to search when looking for executable support
                files (library files and sub-commands).

        Most of the time $Search_Path_Executable are exactly the same.
        However, during "aegis -t -bl" they will be different, with
        $Seach_Path starting at the development directory (the test being run)
        and $Seach_Path_Executable starting at the baseline (the executable
        being run).

   Test Result Codes
        As each test is run (via the test_command field in the project config
        file), aegis determines whether the test succeeded or failed by
        looking at its exit status.  This exit status is mostly as expected
        for UNIX commands.

        Success
            A test should exit 0 to indicate success, i.e. that the specific
            function under test worked as expected.

        Failure
            A test should exit 1 to indicate failure, i.e. that the specific
            function under test did not work as expected.

        No Result
            A test should exit 2 to indicate no result, i.e. that the specific
            function under test could not be exercised because something else
            went wrong.  For example, running out of disk space when creating
            the test input files in the /tmp directory.

        Skipped
            A test should exit 77 to indicate that it was skipped.  This is
            usually to do with the current architecture not being meaningful.
            Whenever possible, use "No Result" instead.  (The value was chosen
            for compatibility with other test systems.)

        Actually, any exit code other than 0, 1 or 77 will be interpreted as
        "no result".  However, always using 0, 1, 2 or 77 means that if a new
        result code is required by a later release of Aegis your existing
        tests will continue to work.

TEST CORRELATIONS

        The "aegis -Test -SUGgest" command may be used to have aegis suggest
        suitable regression tests for your change, based on the source files
        in your change.  This automatically focuses testing effort to relevant
        tests, reducing the number of regression tests necessary to be
        confident that you have not introduced a bug.

        The test correlations are generated by the "aegis -Integrate_Pass"
        command, which associates each test in the change with each source
        file in the change.  Thus, each source file accumulates a list of
        tests which have been associated with it in the past.  This is not as
        exact as code coverage analysis, but is a reasonable approximation in
        practice.

        The aecp(1) and aenf(1) commands are used to associate files with a
        change.  While they do not actively perform the association, these are
        the files used by aeipass(1) and aet(1) to determine which source
        files are associated with which tests.

   Test Correlation Accuracy
        Assuming that the testing correlations are accurate and that the tests
        are evenly distributed across the function space, there will be a less
        than 1/number chance that a relevant test has not been run by the
        "aegis -Test -SUGgest number" command.  A small amount of noise is
        added to the test weighting, so that unexpected things are sometimes
        tested, and the same tests are not run every time.

        Test correlation accuracy can be improved by ensuring that:

        o Each change should be strongly focused, with no gratuitous file
          inclusions.  This avoids spurious correlations.

        o Each item of new functionality should be added in an individual
          change, rather than several together.  This strongly correlates
          tests with functionality.

        o Each bug should be fixed in an individual change, rather than
          several together.  This strongly correlates tests with
          functionality.

        o Test correlations will be lost if files are moved.  This is because
          correlations are by name.

        The best way for tests to correlate accurately with source files is
        when a change contains a test and exactly those files relating to the
        functionality under test.  Too many spurious files will weaken the
        usefulness of the testing correlations.

OPTIONS

        The following options are understood:

        -AUTOmatic
                This option may be used to specify automatic tests.  Automatic
                tests require no human assistance.

        -BaseLine
                This option may be used to specify that the project baseline
                is the subject of the command.

        -BAse_RElative
                This option may be used to cause relative filenames to be
                considered relative to the base of the source tree.  See
                aeuconf(5) for the corresponding user preference.

        -CUrrent_RElative
                This option may be used to cause relative filenames to be
                considered relative to the current directory.  This is usually
                the default.  See aeuconf(5) for the corresponding user
                preference.

        -Change number
                This option may be used to specify a particular change within
                a project.  See aegis(1) for a complete description of this
                option.

        -FOrce  This option may be used to specify that all tests should be
                run, even if the status of the last test run indicates that
                there is no need to run a specific test.

        -Help
                This option may be used to obtain more information about how
                to use the aegis program.

        -INDependent
                This option is used to specify that the test is to be run
                independent of any particular change.  If no tests are named,
                all tests in the baseline will be run.

        -List
                This option may be used to obtain a list of suitable subjects
                for this command.  The list may be more general than expected.

        -MANual This option may be used to specify manual tests.  Manual tests
                require some human intervention,  e.g.: confirmation of some
                screen behavior (X11, for instance), or some user action,
                "unplug ethernet cable now".

        -Not_Logging
                This option may be used to disable the automatic logging of
                output and errors to a file.  This is often useful when
                several aegis commands are combined in a shell script.

        -PErsevere
                This option may be used to specify that all tests should be
                run, even if some fail.  Defaults to the user's
                persevere_preference if not specified, see aeuconf(5) for more
                information.

        -No_PErsevere
                This option may be used to specify that the test run should
                stop after the first failure.  Defaults to the user's
                persevere_preference if not specified, see aeuconf(5) for more
                information.

        -Project name
                This option may be used to select the project of interest.
                When no -Project option is specified, the AEGIS_PROJECT
                environment variable is consulted.  If that does not exist,
                the user's $HOME/.aegisrc file is examined for a default
                project field (see aeuconf(5) for more information).  If that
                does not exist, when the user is only working on changes
                within a single project, the project name defaults to that
                project.  Otherwise, it is an error.

        -PROGress
                This option may be used to specify that progress messages
                should be issued before each test run or before each batch
                test run in case batch_test_command field specified in project
                config file (see aeuconf(5) for more information).

        -No_PROGress
                This option may be used to specify that progress messages
                should be suppressed.  This is the default.

        -REGression
                This option is used to specify that the regression test suite
                is to be run.  The regression test suite consists of all tests
                in the baseline which do not appear in the change.  It is an
                error if there are no regression tests.  You may not name
                tests on the command line when using the -REGression option.
                You may name individual tests to be run on the command line,
                without using the -REGression option; if they are not part of
                the change, the tests of the same name in the baseline will be
                run.

        -SUGgest [ number ]
                The "aegis -Integrate_Pass" command collects test correlation
                statistics when changes are integrated.  This option may be
                used to request that aegis suggest which tests should be run,
                using these testing correlations.  If no number is specified,
                10 tests will be suggested.  This option implies the
                -REGression option.

        -SUGgest_Limit minutes
                This option may be used to limit the number of tests to a
                certain number of minutes.  They will be run from most
                relevant to least relevant.

        -SUGgest_Noise number
                This option may be used to control the amount of noise
                injected into the test selection performed by the -SUGgest
                option.  The number is a percentage of noise to be injected.
                Defaults to 10 if not specified.  The injection of noise
                ensures that a variety of tests are run on subsequent runs,
                and also some from left-field as a sanity check.

        -TERse
                This option may be used to cause listings to produce the bare
                minimum of information.  It is usually useful for shell
                scripts.

        -Verbose
                This option may be used to cause aegis to produce more output.
                By default aegis only produces output on errors.  When used
                with the -List option this option causes column headings to be
                added.

        -Wait   This option may be used to require Aegis commands to wait for
                access locks, if they cannot be obtained immediately.
                Defaults to the user's lock_wait_preference if not specified,
                see aeuconf(5) for more information.

        -No_Wait
                This option may be used to require Aegis commands to emit a
                fatal error if access locks cannot be obtained immediately.
                Defaults to the user's lock_wait_preference if not specified,
                see aeuconf(5) for more information.

        See also aegis(1) for options common to all aegis commands.

        All options may be abbreviated; the abbreviation is documented as the
        upper case letters, all lower case letters and underscores (_) are
        optional.  You must use consecutive sequences of optional letters.

        All options are case insensitive, you may type them in upper case or
        lower case or a combination of both, case is not important.

        For example: the arguments "-project, "-PROJ" and "-p" are all
        interpreted to mean the -Project option.  The argument "-prj" will not
        be understood, because consecutive optional characters were not
        supplied.

        Options and other command line arguments may be mixed arbitrarily on
        the command line, after the function selectors.

        The GNU long option names are understood.  Since all option names for
        aegis are long, this means ignoring the extra leading '-'.  The
        "--option=value" convention is also understood.

RECOMMENDED ALIAS

        The recommended alias for this command is
        csh%    alias aet 'aegis -t \!* -v'
        sh$     aet(){aegis -t "$@" -v}

ERRORS

        It is an error if the change is not in one of the being developed or
        being integrated states.
        It is an error if the change is not assigned to the current user.
        It is an error if your have no relevant tests and no relevant
        exemption.

EXIT STATUS

        The aegis command will exit with a status of 1 on any error.  The
        aegis command will only exit with a status of 0 if there are no
        errors.

ENVIRONMENT VARIABLES

        See aegis(1) for a list of environment variables which may affect this
        command.  See aepconf(5) for the project configuration file's
        project_specific field for how to set environment variables for all
        commands executed by Aegis.

SEE ALSO

        aeb(1)  build a change

        aeca(1) modify the attributes of a change

        aedb(1) begin development of a change

        aeib(1) begin integration of a change

        aent(1) add a new test to a change

        aecp(1) copy an existing test into a change

        aepconf(5)
                project configuration file format

        aeuconf(5)
                user configuration file format

COPYRIGHT

        aegis version 4.24.3.D001
        Copyright (C) 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999,
        2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 Peter
        Miller

        The aegis program comes with ABSOLUTELY NO WARRANTY; for details use
        the 'aegis -VERSion License' command.  This is free software and you
        are welcome to redistribute it under certain conditions; for details
        use the 'aegis -VERSion License' command.

AUTHOR

        Peter Miller   E-Mail:   millerp@canb.auug.org.au
        /\/\*             WWW:   http://www.canb.auug.org.au/~millerp/