Provided by: checkbox-ng_0.23-2_all bug

NAME

       checkbox_ng - CheckboxNG Documentation

       CheckboxNG  is a hardware testing tool useful for certifying laptops, desktops and servers
       with Ubuntu. It is a new version of Checkbox that is built directly on top of PlainBox

       CheckboxNG replaces Checkbox, where applicable.

       WARNING:
          Documentation is under development. Some  things  are  wrong,  inaccurate  or  describe
          development goals rather than current state.

INSTALLATION

       CheckboxNG  can be installed from a PPA (recommended) or pypi on Ubuntu Precise (12.04) or
       newer.

          $ sudo add-apt-repository ppa:checkbox-dev/ppa && sudo apt-get update && sudo apt-get install checkbox-ng

RUNNING STABLE RELEASE UPDATE TESTS

       CheckboxNG has special support for running stable release updates tests  in  an  automated
       manner.  This  runs  all  the  jobs  from  the  sru.whitelist and sends the results to the
       certification website.

       To run SRU tests you will need to know the so-called Secure  ID  of  the  device  you  are
       testing. Once you know that all you need to do is run:

          $ checkbox sru $secure_id submission.xml

       The  second  argument, submission.xml, is a name of the fallback file that is only created
       when sending the data to the certification website fails to work for any reason.

REPORTING BUGS

       To report bugs on the Checkbox project you will need a launchpad  account.  You  may  find
       instructions  on  how  to  create  one <https://help.launchpad.net/YourAccount/NewAccount>
       useful. Once you have an account you can report bugs <https://bugs.launchpad.net/checkbox-
       project/+filebug>.

       On  that  page  you can select the project you wish to file the bug on (we use a number of
       projects to coordinate releases and we prefer to have  bugs  associated  with  appropriate
       part  of Checkbox). If you know the right project to use, just use it and file the bug. If
       you don't know Checkbox internals that much or are in doubt  just  file  it  on  the  base
       'Checkbox'       project       (you       can       use       this       direct       link
       <https://bugs.launchpad.net/checkbox/+filebug>.) A member of  the  development  team  will
       review  your  bug  and  re-assign  it to the appropriate location. The bug number will not
       change when that happens.

THE CHECKBOX STACK

       The Checkbox Stack is a collection of projects that together constitute a complete testing
       and  certification  solution.  It  is composed of the following parts (see table below for
       extra details). All of the projects  are  linked  to  from  the  Launchpad  project  group
       <https://launchpad.net/checkbox-project>.

   Architecture Diagram
       [image: Architecture Diagram] [image]

       This  diagram  contains  a  high-level approximation of the current Checkbox architecture.
       There are three main "pillars". On the left we have end products. Those are  actual  tools
       that certification and engineers are using.  On the right we have the test market. This is
       a open market of tests vendors and suppliers. The tests are wrapped in containers known as
       providers.  In the center we have three shared components. Those implement the bulk of the
       framework and user interfaces for test execution. Finally in the bottom-left corner  there
       is  a  part  of checkbox (a library) that is shared with HEXR for certain tasks. HEXR is a
       out-of-scope web application used by part  of  the  certification  process.  Arrows  imply
       communication with the shape of the arrow shows who calls who.

       As  mentioned  before, in the center column there are three main components of shared code
       (shared by everyone using the end products that are discussed below). The shared  code  is
       composed  of  plainbox,  checkbox-ng  and  checkbox-gui.   Component  responsibilities are
       discussed in more detail in the table below.  Here we can see that checkbox-gui used  DBus
       API  exposed  by  checkbox-ng,  which  in  turn  uses  checkbox-support  (a helper library
       separated out so share some code with HEXR) and plainbox.

       In the right hand side column there are various test providers. The  checkbox  project  is
       producing  and  maintaining a number of providers (see the table below) but it is expected
       that our downstream users will also produce their own providers (specific to a customer or
       project).  Eventually  some  providers  may  come  from  third parties that will adopt the
       format.

       Lastly in the bottom-left corner, the shared library, this library contains  many  parsers
       of  various  file  formats and output formats. Technically this library is a dependency of
       HEXR, checkbox-ng and of providers. As an added complexity the library needs to be  called
       from python3 code and python2 code.

       NOTE:
          The  communication between checkbox-ng and plainbox is bi-directional.  Plainbox offers
          some base interfaces and extension points.  Those  are  all  exposed  through  plainbox
          (using common APIs) but some of those are actually implemented in checkbox-ng.

       WARNING:
          All internal APIs are semi-unstable. The DBus API is more stable in practice but should
          not be relied upon. Projects are encouraged to be merged  into  lp:checkbox  where  API
          transitions  can  be  handled  gracefully.   The  only  stable  API  is are file format
          specification  (job  definitions  and  whitelits).  Launcher  specification   will   be
          stabilized in the next release.

   Component Descriptions
     ┌─────────────────────────┬─────────────────────────────────────────┬──────────────────────────┐
     │Project                  │ Responsible for                         │ Type                     │
     ├─────────────────────────┼─────────────────────────────────────────┼──────────────────────────┤
     │Next Generation Checkbox │                                         │ Application              │
     │(GUI)                    │        • The     C++/QML                │                          │
     │                         │          user interface                 │                          │
     │                         │                                         │                          │
     │                         │        • The   graphical                │                          │
     │                         │          launcher    for                │                          │
     │                         │          providers, e.g.                │                          │
     │                         │          checkbox-certification-client  │                          │
     ├─────────────────────────┼─────────────────────────────────────────┼──────────────────────────┤
     │Next Generation Checkbox │                                         │ Application              │
     │(CLI)                    │        • The    python    command-line  │                          │
     │                         │          interface                      │                          │
     │                         │                                         │                          │
     │                         │          • the text user interface      │                          │
     │                         │                                         │                          │
     │                         │          • the SRU testing command      │                          │
     │                         │                                         │                          │
     │                         │        • Additional certification APIs  │                          │
     │                         │                                         │                          │
     │                         │          • sending data to Launchpad    │                          │
     │                         │                                         │                          │
     │                         │          • sending data to HEXR         │                          │
     │                         │                                         │                          │
     │                         │        • the  DBus  service  needed by  │                          │
     │                         │          GUI                            │                          │
     └─────────────────────────┴─────────────────────────────────────────┴──────────────────────────┘

     │Client     Certification │                                         │ Provider                 │
     │Provider                 │        • canonical-certification-client │                          │
     │                         │          executable                     │                          │
     │                         │                                         │                          │
     │                         │        • client          certification  │                          │
     │                         │          whitelists                     │                          │
     ├─────────────────────────┼─────────────────────────────────────────┼──────────────────────────┤
     │Server     Certification │                                         │ Provider                 │
     │Provider                 │        • server           certification │                          │
     │                         │          whitelists                     │                          │
     │                         │                                         │                          │
     │                         │        • additional server whitelists   │                          │
     ├─────────────────────────┼─────────────────────────────────────────┼──────────────────────────┤
     │System-on-Chip    Server │                                         │ Provider                 │
     │Certification Provider   │        • SoC    server    certification │                          │
     │                         │          whitelists                     │                          │
     ├─────────────────────────┼─────────────────────────────────────────┼──────────────────────────┤
     │Checkbox Provider        │                                         │ Provider                 │
     │                         │        • Almost all job definitions     │                          │
     │                         │                                         │                          │
     │                         │        • Most of custom "scripts"       │                          │
     │                         │                                         │                          │
     │                         │        • Default and SRU whitelist      │                          │
     ├─────────────────────────┼─────────────────────────────────────────┼──────────────────────────┤
     │Resource Provider        │                                         │ Provider                 │
     │                         │        • Almost all resource jobs       │                          │
     │                         │                                         │                          │
     │                         │        • Almost all resource "scripts"  │                          │
     ├─────────────────────────┼─────────────────────────────────────────┼──────────────────────────┤
     │Checkbox Support         │                                         │ Library                  │
     │                         │        • Support   code   for   various │                          │
     │                         │          providers                      │                          │
     │                         │                                         │                          │
     │                         │        • Parsers for many text formats  │                          │
     ├─────────────────────────┼─────────────────────────────────────────┼──────────────────────────┤
     │PlainBox                 │                                         │ Library  and Development │
     │                         │        • Almost all core logic          │ Toolkit                  │
     │                         │                                         │                          │
     │                         │          • RFC822   (job    definition) │                          │
     │                         │            parser                       │                          │
     │                         │                                         │                          │
     │                         │          • Configuration handling       │                          │
     │                         │                                         │                          │
     │                         │          • Testing              session │                          │
     │                         │            (suspend/resume)             │                          │
     │                         │                                         │                          │
     │                         │          • Job runner                   │                          │
     │                         │                                         │                          │
     │                         │          • Trusted launcher             │                          │
     │                         │                                         │                          │
     │                         │          • Dependency resolver          │                          │
     │                         │                                         │                          │
     │                         │          • Command line handling        │                          │
     │                         │                                         │                          │
     │                         │          • The  XML,  HTML   and   XSLX │                          │
     │                         │            exporters                    │                          │
     │                         │                                         │                          │
     │                         │          • and more...                  │                          │
     │                         │                                         │                          │
     │                         │        • Provider development toolkit   │                          │
     │                         │                                         │                          │
     │                         │          • 'plainbox startprovider'     │                          │
     │                         │                                         │                          │
     │                         │          • 'manage.py' implementation   │                          │
     └─────────────────────────┴─────────────────────────────────────────┴──────────────────────────┘

     │Legacy   Checkbox    (no │                                         │ Monolithic   Application │
     │longer maintained)       │        • Applications                   │ Library and Data         │
     │                         │                                         │                          │
     │                         │          • Qt4 GUI                      │                          │
     │                         │                                         │                          │
     │                         │          • Gtk2 GUI                     │                          │
     │                         │                                         │                          │
     │                         │          • Urwid (text) GUI             │                          │
     │                         │                                         │                          │
     │                         │        • Core                           │                          │
     │                         │                                         │                          │
     │                         │          • Plugin  and  Event / Message │                          │
     │                         │            Engine                       │                          │
     │                         │                                         │                          │
     │                         │          • Almost     Every     feature │                          │
     │                         │            implemented a core plugin    │                          │
     │                         │                                         │                          │
     │                         │        • Data                           │                          │
     │                         │                                         │                          │
     │                         │          • Jobs and whitelists          │                          │
     └─────────────────────────┴─────────────────────────────────────────┴──────────────────────────┘

CHANGELOG

       NOTE:
          This changelog contains only a summary of changes. For a more  accurate  accounting  of
          development history please inspect the source history directly.

   CheckboxNG 0.23 (unreleased)
       • Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.23

   CheckboxNG 0.22
       • Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.22

   CheckboxNG 0.3
       • Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.3

   CheckboxNG 0.2
       • Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.2

   CheckboxNG 0.1
       • Initial release

       • Support for displaying configuration

       • Support for running SRU tests (automatic regression testing)

TEST SCRIPTS

       Test 'scripts' are small programs which are used to aid in implementing tests.

   brightness_test
       This  script  tests  the  brightness  of the systems backlight can be changed by using the
       kernel interfaces in /sys/class/backlight. There may be more than one interface to  choose
       from,  so  the  correct  interface to use is selected by using the heuristic prescribed in
       https://www.kernel.org/doc/Documentation/ABI/stable/sysfs-class-backlight. The  brightness
       is  manipulated by updating the brightness file of the interface and the actual_brightness
       file is checked to see if the value was modified to the brightness selected.

PROFILES CONFIGURATION

       Execution profiles, or launchers, allow  specifying  a  predefined  set  of  configuration
       options  that  allow  customization of the welcome screen, displayed whitelists as well as
       saving results locally or sending the submission file to Launchpad or to the Certification
       database/HEXR, as well as some other parameters.

       The  profile  settings  are  part  of  a  launcher  script  and use either checkbox-gui or
       checkbox-launcher (in text-mode/CLI) as a shebang to interpret the key/values.

       This document provides a reference on launcher functionality and syntax. To understand the
       design  and concepts and see several examples, you may want to read the tutorial on how to
       create launchers and their relationship with legacy Checkbox.

   Syntax
       As checkbox-gui is a Qt application, settings must  follow  the  INI-style  rules  of  the
       QSettings <http://qt-project.org/doc/qt-5/QSettings.html> class.

       Multiple-line  values are supported but must be enclosed in doubles quotes and extra lines
       must start with one space, e.g:

          [category]
          key = "Hello
           World"

       • From QML:

          settings.value("category/key", i18n.tr("default_value"))

       • From C++:

          settings->value("category/key", app.tr("default_value"))

       Conversely,  checkbox-launcher-specific  launchers   must   follow   Python   ConfigParser
       <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure> syntax.

       Also,  some settings only make sense for either GUI or CLI, and are thus not understood by
       the other. These are noted below.

   Supported Settings
       welcome/title
              QML application title and welcome screen header. Defaults to System Testing.

       welcome/text
              Welcome message to display on the first screen  (checkbox-gui  supports  Rich  text
              allowing HTML-style markup). Defaults to <p>Welcome to System Testing.</p> [...]

       suite/whitelist_filter
              Regular  expression  to  match  a subset of whitelist filenames. On checkbox-gui it
              defaults to .*. For checkbox-launcher it has no default and must be defined.

       suite/whitelist_selection
              Pattern that whitelists need to match to be preselected. Python regular expression.
              It has no default and must be defined.  (CLI only)

       suite/skip_whitelist_selection
              If  set  to true, user will not receive a choice of whitelist. Only the preselected
              ones (see whitelist_selection) will be selected.  (CLI only).

       suite/skip_test_selection
              If set to true, user will not be allowed to deselect tests prior to run: all  tests
              in the selected whitelist will be run. (CLI only)

       submission/message
              Header  text  of  the  submission  pop-up  , shown to the user after submission has
              completed. (GUI only)

       submission/input_type
              Show a Text input field to enter the secure ID or the  LP  address  (default).   To
              just  save the results to disk, must use the none value. To validate using a regex,
              must be regex.  (GUI only)

       submission/regex
              Regular expression to validate input in submission field (e.g.   email,  secure_id)
              if input_type is regex. (GUI only).  RegExpValidator, default .*

       submission/input_placeholder
              Temporary  text  to  put  in input field, used to guide the user.  Launchpad E-Mail
              Address (default) or Secure ID (15 or 18 characters). (GUI only)

       submission/secure_id
              Preconfigured secure_id to fill in the text field.

       submission/ok_btn_text
              The label for the "Send" button. Submit Results (default)  or  Save  Results.  (GUI
              only)

       submission/cancel_warning
              Show to the user if he wants to exit without having saved the report. You are about
              to exit this test run without saving your results report. Do you want to  save  the
              report? (GUI only)

       submission/submit_to_hexr
              Boolean,  add  an  extra  header  to  also send the results to HEXR (works with the
              certification transport)

       exporter/xml_export_path
              Location to save the XML submission file, if set to an empty  string  will  open  a
              file save dialog. Default: /tmp/submission.xml (GUI only)

       transport/submit_to
              Transport  endpoint.  Defaults  to <none>.  Supports submission to LP (the default,
              value launchpad), certification, or local (save to disk)

       transport/submit_url
              URL to submit results to. This allows to upload to different websites, for  example
              it  can  upload  directly  to  hexr,  or  to  the staging sites. Used only with the
              certification submit_to value.

       transport/config_filename
              Name of a custom config file to load.  Config  files  are  mainly  used  to  define
              environment variables. (CLI only)

       transport/dont_suppress_output
              If  set,  resources,  local  jobs  and  attachments  will be output to screen, this
              generates a lot of text and is mainly for debugging.  (CLI only)

CHECKBOX/PLAINBOX LAUNCHERS TUTORIAL

       This document provides an explanation of why launchers are necessary, what you can achieve
       with  them,  and  goes  over several examples to better describe their capabilities. For a
       detailed reference on which settings are supported by launchers, and specific  syntax  for
       launcher files, look at /profiles.

   Legacy checkbox behavior control
       In the past, Checkbox's behavior was controlled by three mechanisms.

       First,  the  functions of checkbox could be augmented by adding plugins.  For example, the
       ability to submit to certification website was added by the checkbox-certification package
       using  a plugin. The plugins included by checkbox-certification and which add new behavior
       to base checkbox were:

          /usr/share/checkbox-certification/plugins/certify_message.py
          /usr/share/checkbox-certification/plugins/submission_info.py
          /usr/share/checkbox-certification/plugins/backup.py
          /usr/share/checkbox-certification/plugins/certify_prompt.py
          /usr/share/checkbox-certification/plugins/certify_report.py
          /usr/share/checkbox-certification/plugins/certify_schemas.py

       These added the way to prompt the user for  submission-specific  data,  generate  the  xml
       report, and other functions.

       Next,  the  behaviors of the plugins could be configured or controlled using configuration
       files, which are "cascaded". A config file can  include  others  and  those  can  in  turn
       include others.

       This  is  an example of a project-specific project-qt.ini main config file. It's the first
       file read when the project-specific client is launched. Some settings are abbreviated:

          [DEFAULT]
          includes = %(checkbox_oem_share)s/configs/checkbox-project-base-qt.ini %(checkbox_project_share)s/configs/checkbox-project-base.ini

          [checkbox/plugins/environment_info]
          repositories = deb http://.*\(archive\|security\).ubuntu.com/ubuntu precise-security
          routers = multiple
          server_iperf = 10.20.30.40
          sources_list = /etc/apt/sources.list
          wpa_n_psk = password
          wpa_n_ssid = access-point

          [checkbox/plugins/user_interface]
          title = My project System Testing

       Notice  the  includes  line,  this  instructs   it   to   load   the   config   file   for
       checkbox-project-base-qt  and  checkbox-project-base.  Checkbox-project-base-qt  loads the
       configs for checkbox-certification and checkbox-project.  Settings  are  cascaded  so  the
       config options near the top override the ones near the bottom.

       Finally, the "binary" used to invoke checkbox is a shell script that defines where to find
       the things checkbox needs to run: you can  define  a  share  directory,  a  specific  data
       directory,  point  to  a configuration file and define some environment variables that you
       may need during testing. Here's an example for checkbox-project-qt:

          #!/bin/bash
          export CHECKBOX_DATA=${CHECKBOX_DATA:-~/.checkbox}
          export CHECKBOX_SHARE=${CHECKBOX_SHARE:-/usr/share/checkbox}
          export CHECKBOX_OPTIONS=${CHECKBOX_OPTIONS:---log-level=debug --log=$CHECKBOX_DATA/checkbox-project.log}
          export CHECKBOX_CERTIFICATION_SHARE=${CHECKBOX_CERTIFICATION_SHARE:-/usr/share/checkbox-certification}
          export CHECKBOX_OEM_SHARE=${CHECKBOX_PROJECT_BASE_SHARE:-/usr/share/checkbox-project-base}
          export CHECKBOX_PROJECT_SHARE=${CHECKBOX_PROJECT_SHARE:-/usr/share/checkbox-project}

          # Convenience for defining the PYTHONPATH directory.
          if [ "$CHECKBOX_SHARE" != "/usr/share/checkbox" ]; then
              export PYTHONPATH="$CHECKBOX_SHARE:$PYTHONPATH"
          fi

          python3 $CHECKBOX_SHARE/run "$@" $CHECKBOX_PROJECT_SHARE/configs/$(basename $0).ini

       Here you can see that it defines some locations and an important part is the final python3
       line, where it will locate and use the required .ini config file we saw earlier.

       This  hierarchical  organization  was  very powerful but was also difficult to handle, and
       also had some limitations. Part of the work we did with checkbox was to integrate all  the
       project-specific  plugins into checkbox trunk, this way all the core code is in one place,
       and the project-specific variants only supply jobs, whitelists,  data  and  configuration,
       without adding new behavior.

   New plainbox behavior control
       Unlike  checkbox,  plainbox's  core  is monolythic, and it has no concept of plugins. This
       makes it easier to understand and work with. The plainbox core has implementations for all
       the  functions in the old checkbox packages, so no additions are necessary to use features
       such as submission to certification or report generation.

       What we call plainbox is the library that implements all the functionality, as can be seen
       here.

       Plainbox  provides  tools  to  help  test  developers  write and package tests.  These are
       delivered in "providers", which are entities designed to  encapsulate  test  descriptions,
       custom  scripts  for  testing,  whitelists  and assorted data. They were designed to allow
       teams to write and deliver  their  custom  tests  without  worrying  too  much  about  the
       underlying plainbox code.

       To get information on how to write tests and providers, see the Provider Tutorial

       However,  when  actually  using  these tests to verify a real system, we wanted to provide
       something easier and closer to the user experience of checkbox. We  created  two  clients,
       checkbox-gui  and  checkbox-cli,  which  had some hardcoded behaviors, and we also started
       creating other clients which were based on these but were purpose specific. For  instance,
       we had a version of checkbox for SRU testing, another for server certification, and so on.

       But  then  we realized that a lot of the code was duplicated and the behaviors were common
       except for a few changes. So we came  up  with  the  concept  of  "launchers",  which  are
       somewhat similar to checkbox's configuration files and shell script launchers.

       The  idea is that checkbox-gui and checkbox-cli have some very basic behaviors, since they
       are the clients that are shipped by default with ubuntu. They can show all  the  available
       whitelists,  show  a  predefined welcome message, and at the end will let the user see the
       html report and submit it to launchpad using their e-mail address, similar to the  version
       of checkbox that shipped with Ubuntu.

       Instead  of using complicated command line switches, launchers allow you to configure some
       optional behaviors to customize your testing experience. A launcher contains settings, and
       is  similar  to  a  shell  script,  but  the  interpreter  will  be either checkbox-gui or
       checkbox-launcher.

       Here are a few examples of what can be done with launchers.

       As a surprise, checkbox-cli is itself a launcher:

          #!/usr/bin/env checkbox-launcher
          [welcome]
          text = Welcome to System Testing!
              Checkbox provides tests to confirm that your system is working properly.
              Once you are finished running the tests, you can view a summary report for
              your system.
              Warning: Some tests could cause your system to freeze or become
              unresponsive. Please save all your work and close all other running
              applications before beginning the testing process.

          [suite]
          whitelist_filter = ^default$
          whitelist_selection = ^default$
          skip_whitelist_selection = True

          [transport]
          submit_to = launchpad

       You can see here we customize a few options: it shows  a  welcome  message,  automatically
       selects the default whitelist, and will submit to launchpad when it's finished.

       A graphical launcher example is canonical-certification-client.

          #!/usr/bin/checkbox-gui

          [welcome]
          title = "System Certification"
          text = "<p>Welcome to System Certification!</p><p></p><p>This application will
          gather information from your system. Then you will be asked manual tests to
          confirm that the system is working properly. Finally, you will be asked for
          the Secure ID of the computer to submit the information to the certification
          database.</p><p></p><p> To learn how to create or locate the Secure ID,
          please see here: <a href=\"https://certification.canonical.com\">certification.canonical.com</a></p><p></p>"

          [suite]
          whitelist_filter = "^client-(cert|selftest).*"

          [submission]
          input_type = "regex"
          input_placeholder = "Secure ID (15 or 18 characters)"
          ok_btn_text = "Submit Results"
          submit_to_hexr = "true"

          [exporter]
          xml_export_path = "/tmp/submission.xml"

          [transport]
          submit_to = "certification"

       Graphical  launchers  are  a  bit  more complicated, but essentially it's similar, what it
       allows is for you to define some parameters to customize your testing experience.

       A very simple text-mode launcher is canonical-hw-collection  which  just  runs  the  basic
       hardware information tests and uploads them to a hardware database:

          [welcome]
          title = Gathering hardware information
          text = Gathering hardware information.  You may be prompted for your password.
                 This process will take approximately 30 seconds and you will be provided
                 with a URL through which you can confirm and register your hardware
                 submission.

          [suite]
          whitelist_filter = ^hwsubmit$
          whitelist_selection = ^hwsubmit$
          skip_whitelist_selection = True
          skip_test_selection = True

          [submission]
          # A bogus secure_id ensures we don't ask it
          # It can always be overridden in the .conf file.
          secure_id = 000

          [transport]
          submit_to = certification
          submit_url = https://hardware-server.example.com/

       FInally,  canonical-driver-test-suite  provides both a graphical and a text mode launcher,
       which are functionally equivalent:

          #!/usr/bin/checkbox-gui

          [welcome]
          title = "Canonical Driver Test Suite"
          text = "<p>Welcome to the Canonical Driver Test Suite.</p>
           <p></p>
           <p>This program contains automated and manual tests to help you discover
           issues that will arise when running your device drivers on Ubuntu.</p>
           <p></p>
           <p>This application will step the user through these tests in a
           predetermined order and automatically collect both system information as
           well as test results. It will also prompt the user for input when manual
           testing is required.</p>
           <p></p>
           <p>The run time for the tests is determined by which tests you decide to
           execute. The user will have the opportunity to customize the test run to
           accommodate the driver and the amount of time available for testing.</p>
           <p></p>
           <p>To begin, simply click the Continue button below and follow the onscreen
           instructions.</p><p></p>"

          [suite]
          whitelist_filter = "^ihv-.*"

          [submission]
          ok_btn_text = "Save Results"
          input_type = "none"

          [exporter]
          xml_export_path = ""

          [transport]
          submit_to = "local"

       Text mode:

          #!/usr/bin/env checkbox-launcher
          [welcome]
          text = Welcome to the Canonical Driver Test Suite
              This program contains automated and manual tests to help you discover
              issues that will arise when running your device drivers on Ubuntu.
              This application will step the user through these tests in a
              predetermined order and automatically collect both system information as
              well as test results. It will also prompt the user for input when manual
              testing is required.
              The run time for the tests is determined by which tests you decide to
              execute. The user will have the opportunity to customize the test run to
              accommodate the driver and the amount of time available for testing.
              To begin, simply click the Continue button below and follow the onscreen
              instructions.

          [suite]
          # Whitelist(s) displayed in the suite selection screen
          whitelist_filter = ^ihv-.*
          # Whitelist_selection is mandatory so we set it to a bogus value so
          # no whitelists are preselected.
          whitelist_selection = bogus

CHECKBOX RELEASE PROCESS

       This page describes the necessary steps for releasing versions of  Checkbox  and  Checkbox
       Certification to the stable PPA belonging to the Hardware Certification team, on a regular
       basis. Throughout this document the term 'Checkbox' is used as a catch-all term  to  cover
       all  versions  of  Checkbox  owned  by the Hardware Certification team, currently Checkbox
       itself and the Checkbox Certification extensions.

   Overview
       Currently the process runs on a bi-weekly cadence, with a new release  of  Checkbox  every
       two weeks. This covers ten working days, and the tasks carried out on each day or group of
       days is described below:

       • Days 1-4: Time allowed for new changes to be introduced into trunk.

       • Day 5: Changes are merged from the trunk of lp:checkbox and lp:checkbox-certification to
         their respective release branches.  The changelogs for both are bumped at this point and
         revisions are tagged.  At this stage it may also be necessary to copy the package 'fwts'
         from  the  FWTS  Stable  PPA <https://launchpad.net/~firmware-testing-team/+archive/ppa-
         fwts-stable> to  the  Checkbox  Release  Testing  PPA  <https://launchpad.net/~checkbox-
         dev/+archive/testing>.

       • Days  6-9:  Testing  is  performed by the release manager for the Hardware Certification
         team, and a representative of the CE QA team (the  main  customer  for  Checkbox  within
         Canonical)

       • Day  9:  A  release  meeting  is  held  between  the  release  manager  for the Hardware
         Certification team and the representative of the CE QA team.  Potential issues with  the
         release are identified and plans made to address them.

       • Day 10: The tested version of Checkbox is copied to the stable PPA.

   Launchpad Branches
       The  release  process  requires  separate  branches  in Launchpad containing a semi-frozen
       version of the code that was in trunk on day 5 of the process. This is so that development
       can  continue on trunk without jeopardising the stability of the to-be released version of
       Checkbox. The relationship between all branches involved in the process is as shown below:

       • lp:checkbox/release <- lp:checkboxlp:checkbox-certification/release <- lp:checkbox-certificationlp:~checkbox-dev/checkbox/checkbox-packaging-release                                  <-
         lp:~checkbox-dev/checkbox/checkbox-packaging

   Auditing milestoned bugs
       Prior to creating the release candidate the release manager should review the list of bugs
       milestoned for the next  release  of  Checkbox.  They  should  visit  checkbox  milestones
       <https://launchpad.net/checkbox/+milestonesmilestones> and locate the milestone dated with
       the release date.

       • For bugs that are set to In Progress with a branch associated - liase  with  the  branch
         owner to see if the merge can be completed before the deadline.

       • For  bugs  that  are in any other non-closed status (except Fix Commited) - re-milestone
         them to the following milestone.

   Cutting the release
       In order to cut the release, we have to merge the changes  from  trunk  into  the  release
       branch,  commit  them  with  a  suitable message and update the changelog in trunk so that
       future changes go under the correct version. For each combination of branches shown above,
       do the following (the example uses lp:checkbox and lp:checkbox/release):

          bzr branch lp:checkbox/release checkbox-release
          bzr branch lp:checkbox checkbox-trunk
          cd checkbox-release
          current_stable=`head -n1 $(find . -name 'changelog') | grep -oP '(?<=\().*(?=\))'`
          bzr merge lp:checkbox

       at  this point if no changes (other than one to debian/changelog) get merged in then we do
       not perform a release of the package in question. In  practice  this  often  happens  with
       checkbox-certification but never with checkbox:

          bzr commit -m "Merged in changes from rev$(bzr revno -r tag:$current_stable lp:checkbox) to rev$(bzr revno lp:checkbox) from lp:checkbox"
          bzr push lp:checkbox/release
          cd `find . -name 'debian'`; cd ..
          bzr tag `head -n1 debian/changelog | grep -oP '(?<=\().*(?=\))'`
          dch -r (save modified changelog)
          dch -i -U 'Incremented changelog'
          debcommit
          bzr push lp:checkbox

       The   last   step  in  the  process  is  to  perform  a  build  of  the  packages  in  the
       ppa:checkbox-dev/testing PPA. To do this we need  to  go  to  the  recipe  pages  for  the
       checkbox and/or checkbox-certification release branches.

          • checkbox-testing  recipe  <https://code.launchpad.net/~checkbox-dev/+recipe/checkbox-
            testing>

          • checkbox-certification-testing     recipe      <https://code.launchpad.net/~checkbox-
            dev/+recipe/checkbox-certification-testing>

       The Build Now option should be available on the page. Click it to start a build.

   Copying Firmware Test Suite to the Testing PPA
       The  Firmware Test Suite tool is a test tool for system firmware that is naturally heavily
       utilised by Checkbox. To make sure  the  latest  version  which  contains  fixes  and  new
       tests/features  needed  by  Checkbox  is  available  and  also  doesn't  break anything in
       Checkbox, we need to release it alongside Checkbox.  After  cutting  the  release  if  the
       Firmware  Testing team have notified that a new version is available and that this version
       should be used for certification, we need to copy it to the Testing PPA.  To  do  this  we
       need  to  go  to  the  Copy  packages  view  of  the  Firmware  Test  Suite  (Stable)  PPA
       <https://launchpad.net/~firmware-testing-team/+archive/ppa-fwts-stable/+copy-packages> and
       select  the  'fwts'  packages  for  all  releases  back  to  Precise.  We  need to set the
       'Destination PPA' as 'Checkbox Release  Testing  [~checkbox-dev/testing]'  and  the  'Copy
       options'  field  to  'Copy  existing binaries', then click 'Copy packages'. This step then
       needs to be repeated but set the 'Destination PPA' field to 'PPA for  Checkbox  Developers
       [~checkbox-dev/ppa]'.

   Next Release of Checkbox e-mail
       So  that  everyone has the opportunity to perform whatever testing is required in a timely
       manner, after the PPA builds have been completed an email should be sent to the  following
       mailing lists:

       • hardware-certification-team@lists.canonical.com                 <hardware-certification-
         team@lists.canonical.com>

       • commercial-engineering@lists.canonical.com <commercial-engineering@lists.canonical.com>

       The content is typically something like this:

          Subject: Next Release of Checkbox (18/11/2013)

          Hi,

          The next release of Checkbox is available in the
          https://code.launchpad.net/~checkbox-dev/+archive/testing PPA.
          Please test it at your convenience. Checkbox is based on revision 2484 of
          lp:checkbox and Checkbox Certification is based on revision 586 of
          lp:checkbox-certification.

          Thanks,

       If one or the other of Checkbox and Checkbox Certification  have  not  been  updated  then
       there is no need to mention that package

   Testing the release
       Now that the release has been cut, testing should take place prior to the release meeting.
       From  the  point  of  view  of  the  certification  team,  what  needs  to  be  tested  is
       checkbox-certification-client  and  checkbox-certification-server which form the basis for
       CE QAs OEM specific versions of Checkbox. Checkbox certification server is tested  in  the
       CI loop Checkbox certification client needs to be tested manually.

   Release Meeting
       On  the Thursday before the release is made, a meeting is held between a representative of
       the Certification team and a representative of the Commercial  Engineering  QA  team.  The
       meeting    is    held    at    7:30    UTC    as    shown    in   this   calendar   invite
       <https://www.google.com/calendar/hosted/canonical.com/event?action=TEMPLATE&tmeid=Y3QxcWVla3ViMTRvMXByOHZlOTFvc283Y2NfMjAxMzA4MjlUMDczMDAwWiBicmVuZGFuLmRvbmVnYW5AY2Fub25pY2FsLmNvbQ&tmsrc=brendan.donegan%40canonical.com>.
       An agenda for the meeting is included in the invite.

   Publishing the release
       To  publish  the  release  we  simply  need to copy a number of packages from the Checkbox
       Release Testing PPA <https://launchpad.net/~checkbox-dev/+archive/testing> to the Hardware
       Certification  Public PPA <https://launchpad.net/~hardware-certification/+archive/public>.
       To do this we  go  to  the  Copy  packages  view  of  the  Checkbox  Release  Testing  PPA
       <https://launchpad.net/~checkbox-dev/+archive/testing/+copy-packages>   and   select   all
       versions of the following list of packages: checkbox, checkbox-certification,  fwts.  Make
       sure  that  the  'Destination  PPA' field is set to 'Public PPA for Hardware Certification
       [~hardware-certification/public]' and that the  'Copy  options'  field  is  set  to  'Copy
       existing binaries', then click 'Copy Packages'.

       After     that     is    done    an    announcement    email    should    be    sent    to
       commercial-engineering@lists.canonical.com   <commercial-engineering@lists.canonical.com>.
       A template for the announcement in included below:

          Hi,

          A new release of checkbox has been uploaded to the Hardware
          Certification Public PPA
          (https://launchpad.net/~hardware-certification/+archive/public). The
          release is based on revision 2294 of lp:checkbox

          Thanks,

       Please attach the most recent part of the changelog as release notes

       • genindex

       • modindex

       • search

AUTHOR

       Zygmunt Krynicki

COPYRIGHT

       2013-2014, Zygmunt Krynicki