Provided by: plainbox_0.25-1_all bug

NAME

       plainbox-test-plan-units - syntax and semantics of Plainbox test plan unit type

SYNOPSIS

       This page documents the Plainbox test plan units syntax and runtime behavior

DESCRIPTION

       The  test plan unit is an evolution of the Plainbox whitelist concept, that is, a facility
       that describes a sequence of job definitions that should be executed together.

       As in whitelists, jobs definitions are _selected_ by either listing their identifier or  a
       regular  expression  that  matches  their  identifier.  Selected  jobs are executed in the
       sequence they appear in the list, unless they need to be reordered to satisfy dependencies
       which always take priority.

       Unlike  whitelists,  test  plans  can  contain additional meta-data which can be used in a
       graphical user interface. You can assign a translatable name and description to each  test
       plan.  This  used  to be done informally by naming the .whitelist file appropriately, with
       some unique filename and including some #-based comments at the top of the file.

       Test plans are also typical units so they can be defined  with  the  familiar  RFC822-like
       syntax  that  is  also  used  for  job  definitions.  They  can also be multiple test plan
       definitions per file, just like with all the other units, including job definitions.

   Test Plan Fields
       The following fields can be used in a test plan. Note that not all fields need to be  used
       or  even  should  be  used.  Please  remember  that  Checkbox  needs to maintain backwards
       compatibility so some of the  test  plans  it  defines  may  have  non-typical  constructs
       required to ensure proper behavior. You don't have to copy such constructs when working on
       a new test plan from scratch

       id:    Each test plan needs to have a unique identifier. This is exactly the same as  with
              other units that have an identifier (like job definitions and categories).

              This  field  is  not  used  for display purposes but you may need to refer to it on
              command line so keeping it descriptive is useful

       name:  A human-readable name of the test plan. The name should be relatively short  as  it
              may be used to display a list of test plans to the test operator.

              Remember  that  the  user  or the test operator may not always be familiar with the
              scope of testing that you  are  focusing  on.  Also  consider  that  multiple  test
              providers  may  be always installed at the same time. The translated version of the
              name (and icon, see below) is the only thing that needs to allow the test  operator
              to  pick the right test plan.

              Please use short and concrete names like:

                     • "Storage Device Certification Tests"

                     • "Ubuntu Core Application's Clock Acceptance Tests"

                     • "Default Ubuntu Hardware Certification Tests".

              The  field  has  a  soft limit of eighty characters. It cannot have multiple lines.
              This field should be marked as translatable by prepending the underscore  character
              (_) in front. This field is mandatory.

       description:
              A human-readable description of this test plan. Here you can include as many or few
              details as you'd like. Some applications may offer a way of viewing this  data.  In
              general  it is recommended to include a description of what is being tested so that
              users can make an informed decision but please in mind that the  name  field  alone
              must be sufficient to discriminate between distinct test plans so you don't have to
              duplicate that information in the description.

              If your tests will require any special set-up (procuring external hardware, setting
              some  devices  or  software in special test mode) it is recommended to include this
              information here.

              The field has no size limit. It can contain newline characters. This  field  should
              be marked as translatable by prepending the underscore character (_) in front. This
              field is optional.

       include:
              A multi-line list of job identifiers or patterns  matching  such  identifiers  that
              should be included for execution.

              This  is  the  most important field in any test plan. It basically decides on which
              job definitions are selected by (included by) the test plan.  Separate entries need
              to  be  placed  on  separate lines. White space does not separate entries as the id
              field may (sic!) actually include spaces.

              You have two options for selecting tests:

                 • You can simply list the identifier (either partial or fully qualified) of  the
                   job  you  want  to  include in the test plan directly. This is very common and
                   most test plans used by Checkbox actually look like that.

                 • You can use regular expressions to select many tests at the same  time.   This
                   is  the only way to select generated jobs (created either by template units or
                   by job definitions using the legacy 'local' plugin type). Please remember that
                   the  dot  character has a special meaning so unless you actually want to match
                   any character escape the dot with the backslash character (\).

              Regardless of if you use patterns or literal job  identifiers  you  can  use  their
              fully  qualified  name  (the  one that includes the namespace they reside in) or an
              abbreviated form. The abbreviated form  is  applicable  for  job  definitions  that
              reside  in  the  same  namespace  (but  not  necessarily  the same provider) as the
              provider that is defining the test plan.

              Plainbox will  catch  incorrect  references  to  unknown  jobs  so  you  should  be
              relatively  safe. Have a look at the examples section below for examples on how you
              can refer to jobs from other providers (you simply use their fully  qualified  name
              for that)

       mandatory_include:
              A  multi-line  list  of  job identifiers or patterns matching such identifiers that
              should always be executed.

              This optional field can be used to specify the jobs that should always  run.   This
              is  particularly useful for specifying jobs that gather vital info about the tested
              system, as it renders imposible to generate a  report  with  no  information  about
              system under test.

              For  example,  session results meant to be sent to the Ubuntu certification website
              must include the special job: miscellanea/submission-resources

              Example:

                 mandatory_include:
                        miscellanea/submission-resources

              Note that mandatory jobs will always be run first (along with their dependant jobs)

       bootstrap_include:
              A multi-line list of job identifiers that should be run first, before the main body
              of  testing  begins.  The job that should be included in the bootstrapping sections
              are the ones generating or helping to generate other jobs.

              Example:

                 bootstrap_include:
                        graphics/generator_driver_version

              Note that each  entry  in  the  bootstrap_include  section  must  be  a  valid  job
              identifier  and  cannot be a regular expression pattern.  Also note that only local
              and resource jobs are allowed in this section.

       exclude:
              A multi-line list of job identifiers or patterns  matching  such  identifiers  that
              should be excluded from execution.

              This  optional  field  can  be  used  to  prevent some jobs from being selected for
              execution. It follows the similarly named  -x command line option to  the  plainbox
              run command.

              This  field  may  be  used  when a general (broad) selection is somehow made by the
              include field and it must be trimmed down  (for  example,  to  prevent  a  specific
              dangerous job from running). It has the same syntax as the include.

              When a job is both included and excluded, exclusion always takes priority.

       category-overrides:
              A multi-line list of category override statements.

              This  optional  field  can  be  used  to  alter the natural job definition category
              association. Currently Plainbox allows each job definition to associate itself with
              at  most one category (see plainbox-category-units(7) and plainbox-job-units(7) for
              details). This is sub-optimal as some tests can be easily assigned equally well  to
              two categories at the same time.

              For  that  reason,  it may be necessary, in a particular test plan, to override the
              natural category association with one that more correctly reflects the purpose of a
              specific job definition in the context of a specific test plan.

              For  example  let's  consider  a  job  definition that tests if a specific piece of
              hardware works correctly after a suspend-resume cycle. Let's assume  that  the  job
              definition   has  a  natural association with the category describing such hardware
              devices. In one test plan, this test will be associated with the  hardware-specific
              category (using the natural association). In a special suspend-resume test plan the
              same job definition can be associated with a special suspend-resume category.

              The actual rules as to when to use category overrides and how to assign  a  natural
              category  to  a  specific test is not documented here. We believe that each project
              should come up with a workflow and semantics that best match its users.

              The syntax of this field is a list of statements defined on separate  lines.   Each
              override statement has the following form:

                 apply CATEGORY-IDENTIFIER to JOB-DEFINITION-PATTERN

              Both 'apply' and 'to' are literal strings. CATEGORY-IDENTIFIER is the identifier of
              a category unit. The JOB-DEFINITION-PATTERN has the  same  syntax  as  the  include
              field  does. That is, it can be either a simple string or a regular expression that
              is being compared to identifiers of all the known job definitions. The pattern  can
              be  either  partially  or  fully  qualified. That is, it may or may not include the
              namespace component of the job definition identifier.

              Overrides are applied in order and the  last  applied  override  is  the  effective
              override in a given test plan. For example, given the following two overrides:

                 apply cat-1 to .*
                 apply cat-2 to foo

              The  job  definition  with  the  partial identifier foo will be associated with the
              cat-2 category.

       estimated_duration:
              An approximate time to execute this test plan, in seconds.

              Since plainbox version 0.24 this field can be expressed in  two  formats.  The  old
              format, a floating point number of seconds is somewhat difficult to read for larger
              values. To avoid mistakes test designers can use the second  format  with  separate
              sections  for  number  of  hours,  minutes  and  seconds.  The  format,  as regular
              expression, is (\d+h)?[: ]*(\d+m?)[: ]*(\d+s)?. The regular expression expresses an
              optional  number  of  hours, followed by the h character, followed by any number of
              spaces or : characters, followed by an optional number of minutes, followed by  the
              m  character,  again  followed by any number of spaces or : characters, followed by
              the number of seconds, ultimately followed by the s character.

              The values can no longer be fractional (you cannot say 2.5m  you  need  to  say  2m
              30s). We feel that sub-second granularity does is too unpredictable to be useful so
              that will not be supported in the future.

              This field is optional. If it is  missing  it  is  automatically  computed  by  the
              identical field that may be specified on particular job definitions.

              Since  sometimes  it  is easier to think in terms of test plans (they are typically
              executed more often than a specific job  definition)  this  estimate  may  be  more
              accurate as it doesn't include the accumulated sum of mis-estimates from all of the
              job definitions selected by a particular test plan.

   Migrating From Whitelists
       Migrating from whitelists is optional but strongly recommended. Whitelists are discouraged
       but  neither deprecated nor unsupported. As we progress on the transition we are likely to
       fully deprecate and subsequently remove the classical form of whitelits (as are  typically
       found in many *.whitelist files).

       The  first  thing  you  need to do is to create a file that will hold your test plans. You
       should put that file in the units/ directory of your provider.

       Note that a file that holds a test plan may also hold any other units.   The  decision  on
       how to structure your provider is up to you and the particular constraints and recommended
       practices of the project you are participating in.

       Having selected an appropriate file simply copy your old whitelist (just one) and paste it
       into the _template_ below:

          unit: test plan
          id: << DERIVE A PROPER IDENTIFIER FROM THE NAME OF THE WHITELIST FILE >>
          _name: << COME UP WITH A PROPER NAME OF THIS TEST PLAN >>
          _description:
              << COME UP WITH A PROPER DESCRIPTION OF THIS TEST PLAN >>
          include:
              << PASTE THE FULL TEXT OF YOUR OLD WHITELIST >>

       Note  that  you  may  also  add  the  estimated_duration  field  but this is not required.
       Sometimes it is easier to provide a rough estimate of a whole test plan rather than having
       to compute it from all the job definitions it selects.

   Examples
       A simple test plan that selects several jobs:

          id: foo-bar-and-froz
          _name: Tests Foo, Bar and Froz
          _description:
              This example test plan selects the following three jobs:
                  - Foo
                  - Bar
                  - Froz
          include:
              foo
              bar
              froz

       A  test  plan  that uses jobs from another provider's namespace in addition to some of its
       own definitions:

          id: extended-tests
          _name: Extended Storage Tests (By Corp Inc.)
          _description:
              This test plan runs an extended set of storage tests, customized
              by the Corp Inc. corporation. In addition to the standard Ubuntu
              set of storage tests, this test plan includes the following tests::

              - Multipath I/O Tests
              - Degraded Array Recovery Tests
          include:
              2013.com.canonical.certification:disk/.*
              multipath-io
              degrade-array-recovery

       A test plan that generates jobs using bootstrap_include section:

          unit: test plan
          id: test-plan-with-bootstrapping
          _name: Tests with a bootstrapping stage
          _description:
              This test plan uses bootstrapping_include field to generate additional
              jobs depending on the output of the generator job.
          include: .*
          bootstrap_include:
              generator

          unit: job
          id: generator
          plugin: resource
          _description: Job that generates Foo and Bar resources
          command:
           echo "my_resource: Foo"
           echo
           echo "my_resource: Bar"

          unit: template
          template-unit: job
          template-resource: generator
          plugin: shell
          estimated_duration: 1
          id: generated_job_{my_resource}
          command: echo {my_resource}
          _description: Job instantiated from template that echoes {my_resource}

       A test plan that marks some jobs as mandatory:

          unit: test plan
          id: test-plan-with-mandatory-jobs
          _name: Test plan with mandatory jobs
          _description:
              This test plan runs some jobs regardless of user selection.
          include:
              Foo
          mandatory_include:
              Bar

          unit: job
          id: Foo
          _name: Foo job
          _description: Job that might be deselected by the user
          plugin: shell
          command: echo Foo job

          unit: job
          id: Bar
          _name: Bar job (mandatory)
          _description: Job that should *always* run
          plugin: shell
          command: echo Bar job

AUTHOR

       Zygmunt Krynicki & Checkbox Contributors

COPYRIGHT

       2012-2014 Canonical Ltd