xenial (1) plainbox-run.1.gz

Provided by: plainbox_0.25-1_all bug

NAME

       plainbox-run - run a test job

SYNOPSIS

          plainbox run [-h] [--non-interactive] [-n] [--dont-suppress-output]
                       [-f FORMAT] [-p OPTIONS] [-o FILE] [-t TRANSPORT]
                       [--transport-where WHERE] [--transport-options OPTIONS]
                       [-T TEST-PLAN-ID] [-i PATTERN] [-x PATTERN] [-w WHITELIST]

DESCRIPTION

       Run a test job

       This  command  runs  zero or more Plainbox jobs as a part of a single session and saves the test results.
       Plainbox will follow the following high-level algorithm during the execution of this command.

       1. Parse command line arguments and look if there's a session that can be resumed (see  RESUMING  below).
          If  so, offer the user a choice to resume that session. If the resume operation fails move to the next
          qualifying session. Finally offer to create a new session.

       2. If the session is being resumed, replay the effects of the session execution from the  on-disk  state.
          This  recreates  generated  jobs and re-introduces the same resources into the session state. In other
          words, no jobs that have run in the past are re-ran.

          If the resumed session was about to execute a job then  offer  to  skip  the  job.  This  allows  test
          operators to skip jobs that have caused the system to crash in the past (e.g. system suspend tests)

          If the session is not being resumed (a new session was created), set the incomplete flag.

       3. Use  the job selection (see SELECTING JOBS below) to derive the run list. This step involves resolving
          job dependencies and reordering jobs if required.

       4. Follow the run list, executing each job in sequence if possible.  Jobs can be inhibited from execution
          by failed dependencies or failed (evaluating to non-True result) resource expressions.

          If  at  any  time a new job is being re-introduced into the system (see GENERATED JOBS below) then the
          loop is aborted and control jumps back to step 3 to re-select jobs. Existing results are not discarded
          so jobs that already have some results are not executed again.

          Before  and  after  executing  any job the session state is saved to disk to allow resuming from a job
          that somehow crashes the system or crashes Plainbox itself.

       5. Remove the incomplete flag.

       6. Export the state of the session to the desired format (see EXPORTING  RESULTS)  and  use  the  desired
          transport to send the results (see TRANSPORTING RESULTS).

       7. Set the submitted flag.

   SELECTING JOBS
       Plainbox  offers  two  mechanisms for selecting jobs. Both can be used at the same time, both can be used
       multiple times.

   Selecting jobs with patterns
       The first mechanism is exposed through the --include-pattern PATTERN command-line  option.  It  instructs
       Plainbox to select any job whose fully-qualified identifier matches the regular expression PATTERN.

       Jobs  selected  this  way will be, if possible, ordered according to the order of command line arguments.
       For example, having the following command line would run the job foo before running the job bar:
          plainbox run -i '.*::foo' -i '.*::bar'

   Selecting jobs with whitelists
       The second mechanism is the --whitelist WHITELIST command-line option.  WhiteLists (or test plans,  which
       is  somewhat  easier  to  relate  to).   Whitelists  are  simple text files composed of a list of regular
       expressions, identical to those that may be passed with the -i option.

       Unlike the -i option though, there are two kinds of whitelists.  Standalone whitelists are not associated
       with  any  Plainbox  Provider.   Such  whitelists  can  be distributed entirely separately from any other
       component and thus have no association with any namespace.

       Therefore, be fully qualified, each pattern must include both the namespace and  the  partial  identifier
       components. For example, this is a valid, fully quallified whitelist:

          2013.com.canonical.plainbox::stub/.*

       It  will  unambiguously select some of the jobs from the special, internal StubBox provider that is built
       into Plainbox. It can be saved under any filename and stored in any directory and it will  always  select
       the same set of jobs.

       In  contrast,  whitelists  that  are  associated  with  a  particular  provider,  by  being stored in the
       per-provider whitelists/ directory, carry an implicit namespace. Such whitelists  are  typically  written
       without mentioning the namespace component.

       For example, the same "stub/.*" pattern can be abbreviated to:

          stub/.*

       Typically  this  syntax  is  used in all whitelists specific to a particular provider unless the provider
       maintainer explicitly wants to include a job from another namespace (for example, one of  the  well-known
       Checkbox job definitions).

   GENERATED JOBS
       Plainbox offers a way to generate jobs at runtime. There are two motivations for this feature.

   Instantiating Tests for Multiple Devices
       The  classic  example  is  to probe the hardware (for example, to enumerate all storage devices) and then
       duplicate each of the store specific tests so that all devices are tested separately.

       At this time jobs can be generated only from jobs using the plugin type local.  Jobs  of  this  kind  are
       expected to print fully conforming job definitions on stdout. Generated jobs cause a few complexities and
       one limitation that is currently enforced is that generated jobs cannot generate additional jobs  if  any
       of the affected jobs need to run as another user.

       Another limitation is that jobs cannot override existing definitions.

   Creating Parent-Child Association
       A  relatively  niche  and  legacy  feature  of generated jobs is to print a verbatim copy of existing job
       definitions from a local job definition named afer a generic testing theme or category. For  example  the
       Checkbox job definition __wireless__ prints, with the help of cat (1), all of the job definitions defined
       in the file wireless.txt.

       This behavior is special-cased not to cause redefinition errors. Instead, existing definitions  gain  the
       via  attribute  that links them to the generator job. This feature is used by derivative application such
       as Checkbox. Plainbox is not using it at this time.

   RESUMING
       Plainbox  offers  a  session  resume  functionality  whereas  a  session  that  was  interrupted  (either
       purposefully or due to a malfunction) can be resumed and effectively continued where it was left off.

       When resuming a session you may be given an option to either re-run, pass, fail or skip the test job that
       was being executed before the session was interrupted. This is intended to handle both normal situations,
       such  as  a  "system  reboot  test"  where it is perfectly fine to "pass" the test without re-running the
       command. In addition it can be used to handle anomalous cases where the machine misbehaves and re-running
       the same test would cause the problem to occur again indefinitely.

   Limitations
       This functionality does not allow to interrupt and resume a test job that is already being executed. Such
       job will be restarted from scratch.

       Plainbox tries to ensure that a single session is consistent and the assumptions that held at  the  start
       of  the  session are maintained at the end. To that end, Plainbox will try to ensure that job definitions
       have not changed between two separate invocations that worked with a single session.  If such a situation
       is detected the session will not be resumed.

   EXPORTING RESULTS
       Plainbox  offers  a way to export the internal state of the session into a more useful format for further
       processing.

   Selecting Exporters
       The exporter can be selected using the --output-format FORMAT command-line option. A  list  of  available
       exporters  (which  may  include  3rd  party  exporters)  can be obtained by passing the --output-format ?
       option.

       Some formats are more useful than others in that they are capable of transferring more  of  the  internal
       state. Depending on your application you may wish to choose the most generic format (json) and process it
       further with additional tools, choose the most basic format (text) just to get a simple  summary  of  the
       results  or  lastly  choose  one  of  the two specialized formats (xml and html) that are specific to the
       Checkbox workflow.

       Out of the box the following exporters are supported:

   html
       This exporter creates a static HTML page with human-readable test report.  It is useful for communicating
       with other humans and since it is entirely standalone and off-line it can be sent by email or archived.

   json
       This  exporter  creates  a JSON document with the internal representation of the session state. It is the
       most versatile exporter and it is useful  and  easy  for  further  processing.  It  is  not  particularly
       human-readable  but  can  be quite useful for high-level debugging without having to use pdb and know the
       internals of Plainbox.

   rfc822
       This exporter creates quasi-RFC822 documents. It is rather limited and not used much. Still,  it  can  be
       useful in some circumstances.

   text
       This  is  the  default exporter. It simply prints a human-readable representation of test results without
       much detail. It discards nearly all of the internal state though.

   xlsx
       This exporter creates a  standalone  .xlsx  (XML  format  for  Microsoft  Excel)  file  that  contains  a
       human-readable  test report. It is quit similar to the HTML report but it is easier to edit. It is useful
       for communicating with other humans and since it is entirely standalone and off-line it can  be  sent  by
       email or archived.

       It depends on python3-xlsxwriter package

   hexr
       This  exporter  creates  a  rather  confusingly named XML document only applicable for internal Canonical
       Hardware Certification Team workflow.

       It is not a generic XML representation of test  results  and  instead  it  carries  quite  a  few  legacy
       constructs  that  are  only  retained  for  compatibility  with other internal tools. If you want generic
       processing look for JSON instead.

   Selecting Exporter Options
       Certain exporters offer a set of options that can further customize the exported data.  A  full  list  of
       options  available  for  each  exporter  can  be  obtained by passing the --output-options ? command-line
       option.

       Options may be specified as a comma-separated list. Some options act as simple flags, other  options  can
       take an argument with the option=value syntax.

       Known exporter options are documented below:

   json
       with-io-log:
              Exported  data  will  include  the  input/output  log associated with each job result. The data is
              included in its native three-tuple form unless one of the squash-io-log or flatten-io-log  options
              are used as well.

              IO  logs  are  representations  of the data produced by the process created from the shell command
              associated with some jobs.

       squash-io-log:
              When used together with with-io-log option it causes Plainbox  to  discard  the  stream  name  and
              time-stamp  and  just  include a list of base64-encoded binary strings. This option is more useful
              for reconstructing simple "log files"

       flatten-io-log:
              When used together with with-io-log option it causes Plainbox to concatenate all of  the  separate
              base64-encoded  records  into  one  large  base64-encoded  binary  string  representing  the whole
              communication that took place.

       with-run-list:
              Exported data will include the run list (sequence of jobs computed from the desired job list).

       with-job-list:
              Exported data will include the full list of jobs known to the system

       with-resource-map:
              Exported data will include the full resource map. Resources are records of key-value sets that are
              associated  with  each job result for jobs that have plugin type resource. They are expected to be
              printed to stdout by such resource jobs and are parsed and stored by Plainbox.

       with-job-defs:
              Exported data will include some of the properties of  each  job  definition.  Currently  this  set
              includes the following fields: plugin, requires, depends, command and description.

       with-attachments:
              Exported  data  will  include  attachments. Attachments are created from stdout stream of each job
              having plugin type attachment. The actual attachments are base64-encoded.

       with-comments:
              Exported data will include comments added by the test operator to each job result that has them.

       with-job-via:
              Exported data will include the via attribute alongside each job result. The via attribute contains
              the  checksum of the job definition that generated a particular job definition. This is useful for
              tracking jobs generated by jobs with the plugin type local.

       with-job-hash:
              Exported data will include the hash attribute alongside each job result. The hash attribute is the
              checksum of the job definition's data. It can be useful alongside with with-job-via.

       machine-json:
              The  generated  JSON  document will be minimal (devoid of any optional whitespace). This option is
              best to be used if the result is not intended to be read by humans as it saves some space.

   rfc822
       All of the options  have  the  same  meaning  as  for  the  json  exporter:  with-io-log,  squash-io-log,
       flatten-io-log,   with-run-list,   with-job-list,   with-resource-map,  with-job-defs,  with-attachments,
       with-comments, with-job-via, with-job-hash.  The only exception is the machine-json option which  doesn't
       exist for this exporter.

   text
       Same as with rfc822.

   xlsx
       with-sys-info:
              Exported  spreadsheet  will  include  a  worksheet  detailing the hardware devices based on lspci,
              lsusb, udev, etc.

       with-summary:
              Exported spreadsheet will include test figures. This includes the percentage of  tests  that  have
              passed, have failed, have been skipped and the total count.

       with-job-description:
              Exported spreadsheet will include job descriptions on a separate sheet

       with-text-attachments:
              Exported spreadsheet will include text attachments on a separate sheet

   xml
       client-name:
              This option allows clients to override the name of the application generating the XML document. By
              default that name is plainbox.  To use this option  pass  --output-options  client-name=other-name
              command-line option.

   TRANSPORTING RESULTS
       Exported  results  can  be  either  saved to a file (this is the most basic, default transport) or can be
       handed to one of the transport systems for further processing. The idea is  that  specialized  users  can
       provide  their  own  transport  systems (often coupled with a specific exporter) to move the test results
       from the system-under-test to a central testing result repository.

       Transport can be selected with the --transport  option.  Again,  as  with  exporters,  a  list  of  known
       transports  can  be obtained by passing the --transport ? option. Transports need a destination URL which
       can be specified with the --transport-where= option. The syntax of the URL varies by transport type.

       Plainbox comes equipped with the following transports:

   launchpad
       This transport can send the results exported using xml exporter to the Launchpad Hardware Database.  This
       is a little-known feature offered by the https://launchpad.net/ website.

   certification
       This  transport  can  send  the  results  exported  using the xml exporter to the Canonical Certification
       Website (https://certification.canonical.com).

       This transport is of little use to anyone  but  the  Canonical  Hardware  Certification  Team  that  also
       maintains Plainbox and Checkbox but it is mentioned here for completeness.

OPTIONS

       Optional arguments:

       --non-interactive
              skip tests that require interactivity

       -n, --dry-run
              don't really run most jobs

       --dont-suppress-output
              don't suppress the output of certain job plugin types

       -f, --output-format
              save test results in the specified FORMAT (pass ? for a list of choices)

       -p, --output-options
              comma-separated list of options for the export mechanism (pass ? for a list of choices)

       -o, --output-file
              save test results to the specified FILE (or to stdout if FILE is -)

       -t, --transport
              use TRANSPORT to send results somewhere (pass ? for a list of choices)

              Possible choices: ?

       --transport-where
              where to send data using the selected transport

       --transport-options
              comma-separated list of key-value options (k=v) to be passed to the transport

       -T, --test-plan
              load the specified test plan

       -i, --include-pattern
              include jobs matching the given regular expression

       -x, --exclude-pattern
              exclude jobs matching the given regular expression

       -w, --whitelist
              load whitelist containing run patterns

SEE ALSO

       plainbox-dev-analyze

AUTHOR

       Zygmunt Krynicki & Checkbox Contributors

       2012-2014 Canonical Ltd