Provided by: plainbox_0.25-1_all 
NAME
plainbox-test-plan-units - syntax and semantics of Plainbox test plan unit type
SYNOPSIS
This page documents the Plainbox test plan units syntax and runtime behavior
DESCRIPTION
The test plan unit is an evolution of the Plainbox whitelist concept, that is, a facility
that describes a sequence of job definitions that should be executed together.
As in whitelists, jobs definitions are _selected_ by either listing their identifier or a
regular expression that matches their identifier. Selected jobs are executed in the
sequence they appear in the list, unless they need to be reordered to satisfy dependencies
which always take priority.
Unlike whitelists, test plans can contain additional meta-data which can be used in a
graphical user interface. You can assign a translatable name and description to each test
plan. This used to be done informally by naming the .whitelist file appropriately, with
some unique filename and including some #-based comments at the top of the file.
Test plans are also typical units so they can be defined with the familiar RFC822-like
syntax that is also used for job definitions. They can also be multiple test plan
definitions per file, just like with all the other units, including job definitions.
Test Plan Fields
The following fields can be used in a test plan. Note that not all fields need to be used
or even should be used. Please remember that Checkbox needs to maintain backwards
compatibility so some of the test plans it defines may have non-typical constructs
required to ensure proper behavior. You don't have to copy such constructs when working on
a new test plan from scratch
id: Each test plan needs to have a unique identifier. This is exactly the same as with
other units that have an identifier (like job definitions and categories).
This field is not used for display purposes but you may need to refer to it on
command line so keeping it descriptive is useful
name: A human-readable name of the test plan. The name should be relatively short as it
may be used to display a list of test plans to the test operator.
Remember that the user or the test operator may not always be familiar with the
scope of testing that you are focusing on. Also consider that multiple test
providers may be always installed at the same time. The translated version of the
name (and icon, see below) is the only thing that needs to allow the test operator
to pick the right test plan.
Please use short and concrete names like:
• "Storage Device Certification Tests"
• "Ubuntu Core Application's Clock Acceptance Tests"
• "Default Ubuntu Hardware Certification Tests".
The field has a soft limit of eighty characters. It cannot have multiple lines.
This field should be marked as translatable by prepending the underscore character
(_) in front. This field is mandatory.
description:
A human-readable description of this test plan. Here you can include as many or few
details as you'd like. Some applications may offer a way of viewing this data. In
general it is recommended to include a description of what is being tested so that
users can make an informed decision but please in mind that the name field alone
must be sufficient to discriminate between distinct test plans so you don't have to
duplicate that information in the description.
If your tests will require any special set-up (procuring external hardware, setting
some devices or software in special test mode) it is recommended to include this
information here.
The field has no size limit. It can contain newline characters. This field should
be marked as translatable by prepending the underscore character (_) in front. This
field is optional.
include:
A multi-line list of job identifiers or patterns matching such identifiers that
should be included for execution.
This is the most important field in any test plan. It basically decides on which
job definitions are selected by (included by) the test plan. Separate entries need
to be placed on separate lines. White space does not separate entries as the id
field may (sic!) actually include spaces.
You have two options for selecting tests:
• You can simply list the identifier (either partial or fully qualified) of the
job you want to include in the test plan directly. This is very common and
most test plans used by Checkbox actually look like that.
• You can use regular expressions to select many tests at the same time. This
is the only way to select generated jobs (created either by template units or
by job definitions using the legacy 'local' plugin type). Please remember that
the dot character has a special meaning so unless you actually want to match
any character escape the dot with the backslash character (\).
Regardless of if you use patterns or literal job identifiers you can use their
fully qualified name (the one that includes the namespace they reside in) or an
abbreviated form. The abbreviated form is applicable for job definitions that
reside in the same namespace (but not necessarily the same provider) as the
provider that is defining the test plan.
Plainbox will catch incorrect references to unknown jobs so you should be
relatively safe. Have a look at the examples section below for examples on how you
can refer to jobs from other providers (you simply use their fully qualified name
for that)
mandatory_include:
A multi-line list of job identifiers or patterns matching such identifiers that
should always be executed.
This optional field can be used to specify the jobs that should always run. This
is particularly useful for specifying jobs that gather vital info about the tested
system, as it renders imposible to generate a report with no information about
system under test.
For example, session results meant to be sent to the Ubuntu certification website
must include the special job: miscellanea/submission-resources
Example:
mandatory_include:
miscellanea/submission-resources
Note that mandatory jobs will always be run first (along with their dependant jobs)
bootstrap_include:
A multi-line list of job identifiers that should be run first, before the main body
of testing begins. The job that should be included in the bootstrapping sections
are the ones generating or helping to generate other jobs.
Example:
bootstrap_include:
graphics/generator_driver_version
Note that each entry in the bootstrap_include section must be a valid job
identifier and cannot be a regular expression pattern. Also note that only local
and resource jobs are allowed in this section.
exclude:
A multi-line list of job identifiers or patterns matching such identifiers that
should be excluded from execution.
This optional field can be used to prevent some jobs from being selected for
execution. It follows the similarly named -x command line option to the plainbox
run command.
This field may be used when a general (broad) selection is somehow made by the
include field and it must be trimmed down (for example, to prevent a specific
dangerous job from running). It has the same syntax as the include.
When a job is both included and excluded, exclusion always takes priority.
category-overrides:
A multi-line list of category override statements.
This optional field can be used to alter the natural job definition category
association. Currently Plainbox allows each job definition to associate itself with
at most one category (see plainbox-category-units(7) and plainbox-job-units(7) for
details). This is sub-optimal as some tests can be easily assigned equally well to
two categories at the same time.
For that reason, it may be necessary, in a particular test plan, to override the
natural category association with one that more correctly reflects the purpose of a
specific job definition in the context of a specific test plan.
For example let's consider a job definition that tests if a specific piece of
hardware works correctly after a suspend-resume cycle. Let's assume that the job
definition has a natural association with the category describing such hardware
devices. In one test plan, this test will be associated with the hardware-specific
category (using the natural association). In a special suspend-resume test plan the
same job definition can be associated with a special suspend-resume category.
The actual rules as to when to use category overrides and how to assign a natural
category to a specific test is not documented here. We believe that each project
should come up with a workflow and semantics that best match its users.
The syntax of this field is a list of statements defined on separate lines. Each
override statement has the following form:
apply CATEGORY-IDENTIFIER to JOB-DEFINITION-PATTERN
Both 'apply' and 'to' are literal strings. CATEGORY-IDENTIFIER is the identifier of
a category unit. The JOB-DEFINITION-PATTERN has the same syntax as the include
field does. That is, it can be either a simple string or a regular expression that
is being compared to identifiers of all the known job definitions. The pattern can
be either partially or fully qualified. That is, it may or may not include the
namespace component of the job definition identifier.
Overrides are applied in order and the last applied override is the effective
override in a given test plan. For example, given the following two overrides:
apply cat-1 to .*
apply cat-2 to foo
The job definition with the partial identifier foo will be associated with the
cat-2 category.
estimated_duration:
An approximate time to execute this test plan, in seconds.
Since plainbox version 0.24 this field can be expressed in two formats. The old
format, a floating point number of seconds is somewhat difficult to read for larger
values. To avoid mistakes test designers can use the second format with separate
sections for number of hours, minutes and seconds. The format, as regular
expression, is (\d+h)?[: ]*(\d+m?)[: ]*(\d+s)?. The regular expression expresses an
optional number of hours, followed by the h character, followed by any number of
spaces or : characters, followed by an optional number of minutes, followed by the
m character, again followed by any number of spaces or : characters, followed by
the number of seconds, ultimately followed by the s character.
The values can no longer be fractional (you cannot say 2.5m you need to say 2m
30s). We feel that sub-second granularity does is too unpredictable to be useful so
that will not be supported in the future.
This field is optional. If it is missing it is automatically computed by the
identical field that may be specified on particular job definitions.
Since sometimes it is easier to think in terms of test plans (they are typically
executed more often than a specific job definition) this estimate may be more
accurate as it doesn't include the accumulated sum of mis-estimates from all of the
job definitions selected by a particular test plan.
Migrating From Whitelists
Migrating from whitelists is optional but strongly recommended. Whitelists are discouraged
but neither deprecated nor unsupported. As we progress on the transition we are likely to
fully deprecate and subsequently remove the classical form of whitelits (as are typically
found in many *.whitelist files).
The first thing you need to do is to create a file that will hold your test plans. You
should put that file in the units/ directory of your provider.
Note that a file that holds a test plan may also hold any other units. The decision on
how to structure your provider is up to you and the particular constraints and recommended
practices of the project you are participating in.
Having selected an appropriate file simply copy your old whitelist (just one) and paste it
into the _template_ below:
unit: test plan
id: << DERIVE A PROPER IDENTIFIER FROM THE NAME OF THE WHITELIST FILE >>
_name: << COME UP WITH A PROPER NAME OF THIS TEST PLAN >>
_description:
<< COME UP WITH A PROPER DESCRIPTION OF THIS TEST PLAN >>
include:
<< PASTE THE FULL TEXT OF YOUR OLD WHITELIST >>
Note that you may also add the estimated_duration field but this is not required.
Sometimes it is easier to provide a rough estimate of a whole test plan rather than having
to compute it from all the job definitions it selects.
Examples
A simple test plan that selects several jobs:
id: foo-bar-and-froz
_name: Tests Foo, Bar and Froz
_description:
This example test plan selects the following three jobs:
- Foo
- Bar
- Froz
include:
foo
bar
froz
A test plan that uses jobs from another provider's namespace in addition to some of its
own definitions:
id: extended-tests
_name: Extended Storage Tests (By Corp Inc.)
_description:
This test plan runs an extended set of storage tests, customized
by the Corp Inc. corporation. In addition to the standard Ubuntu
set of storage tests, this test plan includes the following tests::
- Multipath I/O Tests
- Degraded Array Recovery Tests
include:
2013.com.canonical.certification:disk/.*
multipath-io
degrade-array-recovery
A test plan that generates jobs using bootstrap_include section:
unit: test plan
id: test-plan-with-bootstrapping
_name: Tests with a bootstrapping stage
_description:
This test plan uses bootstrapping_include field to generate additional
jobs depending on the output of the generator job.
include: .*
bootstrap_include:
generator
unit: job
id: generator
plugin: resource
_description: Job that generates Foo and Bar resources
command:
echo "my_resource: Foo"
echo
echo "my_resource: Bar"
unit: template
template-unit: job
template-resource: generator
plugin: shell
estimated_duration: 1
id: generated_job_{my_resource}
command: echo {my_resource}
_description: Job instantiated from template that echoes {my_resource}
A test plan that marks some jobs as mandatory:
unit: test plan
id: test-plan-with-mandatory-jobs
_name: Test plan with mandatory jobs
_description:
This test plan runs some jobs regardless of user selection.
include:
Foo
mandatory_include:
Bar
unit: job
id: Foo
_name: Foo job
_description: Job that might be deselected by the user
plugin: shell
command: echo Foo job
unit: job
id: Bar
_name: Bar job (mandatory)
_description: Job that should *always* run
plugin: shell
command: echo Bar job
AUTHOR
Zygmunt Krynicki & Checkbox Contributors
COPYRIGHT
2012-2014 Canonical Ltd