xenial (1) checkbox.1.gz

Provided by: checkbox-ng_0.23-2_all bug

NAME

       checkbox_ng - CheckboxNG Documentation

       CheckboxNG is a hardware testing tool useful for certifying laptops, desktops and servers with Ubuntu. It
       is a new version of Checkbox that is built directly on top of PlainBox

       CheckboxNG replaces Checkbox, where applicable.

       WARNING:
          Documentation is under development. Some things are wrong, inaccurate or  describe  development  goals
          rather than current state.

INSTALLATION

       CheckboxNG can be installed from a PPA (recommended) or pypi on Ubuntu Precise (12.04) or newer.

          $ sudo add-apt-repository ppa:checkbox-dev/ppa && sudo apt-get update && sudo apt-get install checkbox-ng

RUNNING STABLE RELEASE UPDATE TESTS

       CheckboxNG has special support for running stable release updates tests in an automated manner. This runs
       all the jobs from the sru.whitelist and sends the results to the certification website.

       To run SRU tests you will need to know the so-called Secure ID of the device you are  testing.  Once  you
       know that all you need to do is run:

          $ checkbox sru $secure_id submission.xml

       The second argument, submission.xml, is a name of the fallback file that is only created when sending the
       data to the certification website fails to work for any reason.

REPORTING BUGS

       To report bugs on the Checkbox project you will need a launchpad account. You may  find  instructions  on
       how  to  create  one <https://help.launchpad.net/YourAccount/NewAccount> useful. Once you have an account
       you can report bugs <https://bugs.launchpad.net/checkbox-project/+filebug>.

       On that page you can select the project you wish to file the bug on (we  use  a  number  of  projects  to
       coordinate releases and we prefer to have bugs associated with appropriate part of Checkbox). If you know
       the right project to use, just use it and file the bug. If you don't know Checkbox internals that much or
       are   in   doubt   just  file  it  on  the  base  'Checkbox'  project  (you  can  use  this  direct  link
       <https://bugs.launchpad.net/checkbox/+filebug>.) A member of the development team will  review  your  bug
       and re-assign it to the appropriate location. The bug number will not change when that happens.

THE CHECKBOX STACK

       The  Checkbox  Stack  is  a  collection  of  projects  that  together  constitute  a complete testing and
       certification solution. It is composed of the following parts (see table below for extra details). All of
       the projects are linked to from the Launchpad project group <https://launchpad.net/checkbox-project>.

   Architecture Diagram
       [image: Architecture Diagram] [image]

       This  diagram  contains  a high-level approximation of the current Checkbox architecture. There are three
       main "pillars". On the left we have end products. Those are actual tools that certification and engineers
       are  using.   On the right we have the test market. This is a open market of tests vendors and suppliers.
       The tests are wrapped in containers known as providers. In the center we have  three  shared  components.
       Those  implement  the  bulk  of  the  framework  and  user  interfaces for test execution. Finally in the
       bottom-left corner there is a part of checkbox (a library) that is shared with HEXR  for  certain  tasks.
       HEXR  is  a  out-of-scope  web  application  used  by  part  of  the  certification process. Arrows imply
       communication with the shape of the arrow shows who calls who.

       As mentioned before, in the center column there are three main  components  of  shared  code  (shared  by
       everyone  using  the  end  products  that  are discussed below). The shared code is composed of plainbox,
       checkbox-ng and checkbox-gui.  Component responsibilities are discussed  in  more  detail  in  the  table
       below.   Here  we  can  see  that  checkbox-gui  used DBus API exposed by checkbox-ng, which in turn uses
       checkbox-support (a helper library separated out so share some code with HEXR) and plainbox.

       In the right hand side column there are various test providers. The checkbox  project  is  producing  and
       maintaining a number of providers (see the table below) but it is expected that our downstream users will
       also produce their own providers (specific to a customer or project). Eventually some providers may  come
       from third parties that will adopt the format.

       Lastly  in the bottom-left corner, the shared library, this library contains many parsers of various file
       formats and output formats. Technically this  library  is  a  dependency  of  HEXR,  checkbox-ng  and  of
       providers. As an added complexity the library needs to be called from python3 code and python2 code.

       NOTE:
          The  communication  between  checkbox-ng  and  plainbox  is bi-directional.  Plainbox offers some base
          interfaces and extension points. Those are all exposed through plainbox (using common APIs)  but  some
          of those are actually implemented in checkbox-ng.

       WARNING:
          All  internal APIs are semi-unstable. The DBus API is more stable in practice but should not be relied
          upon. Projects are encouraged to be merged into lp:checkbox  where  API  transitions  can  be  handled
          gracefully.   The  only  stable  API is are file format specification (job definitions and whitelits).
          Launcher specification will be stabilized in the next release.

   Component Descriptions
        ┌─────────────────────────────┬─────────────────────────────────────────┬──────────────────────────────┐
        │Project                      │ Responsible for                         │ Type                         │
        ├─────────────────────────────┼─────────────────────────────────────────┼──────────────────────────────┤
        │Next   Generation   Checkbox │                                         │ Application                  │
        │(GUI)                        │        • The   C++/QML  user            │                              │
        │                             │          interface                      │                              │
        │                             │                                         │                              │
        │                             │        • The       graphical            │                              │
        │                             │          launcher        for            │                              │
        │                             │          providers,     e.g.            │                              │
        │                             │          checkbox-certification-client  │                              │
        ├─────────────────────────────┼─────────────────────────────────────────┼──────────────────────────────┤
        │Next   Generation   Checkbox │                                         │ Application                  │
        │(CLI)                        │        • The    python    command-line  │                              │
        │                             │          interface                      │                              │
        │                             │                                         │                              │
        │                             │          • the text user interface      │                              │
        │                             │                                         │                              │
        │                             │          • the SRU testing command      │                              │
        │                             │                                         │                              │
        │                             │        • Additional certification APIs  │                              │
        │                             │                                         │                              │
        │                             │          • sending data to Launchpad    │                              │
        │                             │                                         │                              │
        │                             │          • sending data to HEXR         │                              │
        │                             │                                         │                              │
        │                             │        • the DBus  service  needed  by  │                              │
        │                             │          GUI                            │                              │
        ├─────────────────────────────┼─────────────────────────────────────────┼──────────────────────────────┤
        │Client         Certification │                                         │ Provider                     │
        │Provider                     │        • canonical-certification-client │                              │
        │                             │          executable                     │                              │
        │                             │                                         │                              │
        │                             │        • client          certification  │                              │
        │                             │          whitelists                     │                              │
        ├─────────────────────────────┼─────────────────────────────────────────┼──────────────────────────────┤
        │Server         Certification │                                         │ Provider                     │
        │Provider                     │        • server           certification │                              │
        │                             │          whitelists                     │                              │
        │                             │                                         │                              │
        │                             │        • additional server whitelists   │                              │
        └─────────────────────────────┴─────────────────────────────────────────┴──────────────────────────────┘

        │System-on-Chip        Server │                                         │ Provider                     │
        │Certification Provider       │        • SoC    server    certification │                              │
        │                             │          whitelists                     │                              │
        ├─────────────────────────────┼─────────────────────────────────────────┼──────────────────────────────┤
        │Checkbox Provider            │                                         │ Provider                     │
        │                             │        • Almost all job definitions     │                              │
        │                             │                                         │                              │
        │                             │        • Most of custom "scripts"       │                              │
        │                             │                                         │                              │
        │                             │        • Default and SRU whitelist      │                              │
        ├─────────────────────────────┼─────────────────────────────────────────┼──────────────────────────────┤
        │Resource Provider            │                                         │ Provider                     │
        │                             │        • Almost all resource jobs       │                              │
        │                             │                                         │                              │
        │                             │        • Almost all resource "scripts"  │                              │
        ├─────────────────────────────┼─────────────────────────────────────────┼──────────────────────────────┤
        │Checkbox Support             │                                         │ Library                      │
        │                             │        • Support   code   for   various │                              │
        │                             │          providers                      │                              │
        │                             │                                         │                              │
        │                             │        • Parsers for many text formats  │                              │
        ├─────────────────────────────┼─────────────────────────────────────────┼──────────────────────────────┤
        │PlainBox                     │                                         │ Library   and    Development │
        │                             │        • Almost all core logic          │ Toolkit                      │
        │                             │                                         │                              │
        │                             │          • RFC822    (job   definition) │                              │
        │                             │            parser                       │                              │
        │                             │                                         │                              │
        │                             │          • Configuration handling       │                              │
        │                             │                                         │                              │
        │                             │          • Testing              session │                              │
        │                             │            (suspend/resume)             │                              │
        │                             │                                         │                              │
        │                             │          • Job runner                   │                              │
        │                             │                                         │                              │
        │                             │          • Trusted launcher             │                              │
        │                             │                                         │                              │
        │                             │          • Dependency resolver          │                              │
        │                             │                                         │                              │
        │                             │          • Command line handling        │                              │
        │                             │                                         │                              │
        │                             │          • The   XML,   HTML  and  XSLX │                              │
        │                             │            exporters                    │                              │
        │                             │                                         │                              │
        │                             │          • and more...                  │                              │
        │                             │                                         │                              │
        │                             │        • Provider development toolkit   │                              │
        │                             │                                         │                              │
        │                             │          • 'plainbox startprovider'     │                              │
        │                             │                                         │                              │
        │                             │          • 'manage.py' implementation   │                              │
        └─────────────────────────────┴─────────────────────────────────────────┴──────────────────────────────┘

        │Legacy  Checkbox  (no longer │                                         │ Monolithic       Application │
        │maintained)                  │        • Applications                   │ Library and Data             │
        │                             │                                         │                              │
        │                             │          • Qt4 GUI                      │                              │
        │                             │                                         │                              │
        │                             │          • Gtk2 GUI                     │                              │
        │                             │                                         │                              │
        │                             │          • Urwid (text) GUI             │                              │
        │                             │                                         │                              │
        │                             │        • Core                           │                              │
        │                             │                                         │                              │
        │                             │          • Plugin and Event  /  Message │                              │
        │                             │            Engine                       │                              │
        │                             │                                         │                              │
        │                             │          • Almost     Every     feature │                              │
        │                             │            implemented a core plugin    │                              │
        │                             │                                         │                              │
        │                             │        • Data                           │                              │
        │                             │                                         │                              │
        │                             │          • Jobs and whitelists          │                              │
        └─────────────────────────────┴─────────────────────────────────────────┴──────────────────────────────┘

CHANGELOG

       NOTE:
          This  changelog  contains  only  a  summary  of changes. For a more accurate accounting of development
          history please inspect the source history directly.

   CheckboxNG 0.23 (unreleased)
       • Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.23

   CheckboxNG 0.22
       • Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.22

   CheckboxNG 0.3
       • Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.3

   CheckboxNG 0.2
       • Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.2

   CheckboxNG 0.1
       • Initial release

       • Support for displaying configuration

       • Support for running SRU tests (automatic regression testing)

TEST SCRIPTS

       Test 'scripts' are small programs which are used to aid in implementing tests.

   brightness_test
       This script tests the brightness of the systems backlight can be changed by using the  kernel  interfaces
       in /sys/class/backlight. There may be more than one interface to choose from, so the correct interface to
       use       is       selected       by        using        the        heuristic        prescribed        in
       https://www.kernel.org/doc/Documentation/ABI/stable/sysfs-class-backlight.  The brightness is manipulated
       by updating the brightness file of the interface and the actual_brightness file is checked to see if  the
       value was modified to the brightness selected.

PROFILES CONFIGURATION

       Execution  profiles,  or launchers, allow specifying a predefined set of configuration options that allow
       customization of the welcome screen, displayed whitelists as well as saving results  locally  or  sending
       the submission file to Launchpad or to the Certification database/HEXR, as well as some other parameters.

       The  profile  settings are part of a launcher script and use either checkbox-gui or checkbox-launcher (in
       text-mode/CLI) as a shebang to interpret the key/values.

       This document provides a reference on launcher functionality and syntax. To  understand  the  design  and
       concepts and see several examples, you may want to read the tutorial on how to create launchers and their
       relationship with legacy Checkbox.

   Syntax
       As checkbox-gui is a  Qt  application,  settings  must  follow  the  INI-style  rules  of  the  QSettings
       <http://qt-project.org/doc/qt-5/QSettings.html> class.

       Multiple-line values are supported but must be enclosed in doubles quotes and extra lines must start with
       one space, e.g:

          [category]
          key = "Hello
           World"

       • From QML:

          settings.value("category/key", i18n.tr("default_value"))

       • From C++:

          settings->value("category/key", app.tr("default_value"))

       Conversely,     checkbox-launcher-specific     launchers     must     follow     Python      ConfigParser
       <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure> syntax.

       Also,  some  settings  only  make  sense for either GUI or CLI, and are thus not understood by the other.
       These are noted below.

   Supported Settings
       welcome/title
              QML application title and welcome screen header. Defaults to System Testing.

       welcome/text
              Welcome message to  display  on  the  first  screen  (checkbox-gui  supports  Rich  text  allowing
              HTML-style markup). Defaults to <p>Welcome to System Testing.</p> [...]

       suite/whitelist_filter
              Regular  expression  to  match a subset of whitelist filenames. On checkbox-gui it defaults to .*.
              For checkbox-launcher it has no default and must be defined.

       suite/whitelist_selection
              Pattern that whitelists need to match to be preselected. Python  regular  expression.  It  has  no
              default and must be defined.  (CLI only)

       suite/skip_whitelist_selection
              If  set  to  true,  user  will  not  receive a choice of whitelist. Only the preselected ones (see
              whitelist_selection) will be selected.  (CLI only).

       suite/skip_test_selection
              If set to true, user will not be allowed to deselect tests prior to run: all tests in the selected
              whitelist will be run. (CLI only)

       submission/message
              Header  text  of  the  submission  pop-up , shown to the user after submission has completed. (GUI
              only)

       submission/input_type
              Show a Text input field to enter the secure ID or the LP address  (default).   To  just  save  the
              results to disk, must use the none value. To validate using a regex, must be regex.  (GUI only)

       submission/regex
              Regular expression to validate input in submission field (e.g.  email, secure_id) if input_type is
              regex. (GUI only).  RegExpValidator, default .*

       submission/input_placeholder
              Temporary text to put in input field, used to guide the user.  Launchpad E-Mail Address  (default)
              or Secure ID (15 or 18 characters). (GUI only)

       submission/secure_id
              Preconfigured secure_id to fill in the text field.

       submission/ok_btn_text
              The label for the "Send" button. Submit Results (default) or Save Results. (GUI only)

       submission/cancel_warning
              Show  to  the user if he wants to exit without having saved the report. You are about to exit this
              test run without saving your results report. Do you want to save the report? (GUI only)

       submission/submit_to_hexr
              Boolean, add an extra header to also send the  results  to  HEXR  (works  with  the  certification
              transport)

       exporter/xml_export_path
              Location  to save the XML submission file, if set to an empty string will open a file save dialog.
              Default: /tmp/submission.xml (GUI only)

       transport/submit_to
              Transport endpoint. Defaults to <none>.  Supports submission to LP (the default, value launchpad),
              certification, or local (save to disk)

       transport/submit_url
              URL  to  submit results to. This allows to upload to different websites, for example it can upload
              directly to hexr, or to the staging sites. Used only with the certification submit_to value.

       transport/config_filename
              Name of a custom config file  to  load.  Config  files  are  mainly  used  to  define  environment
              variables. (CLI only)

       transport/dont_suppress_output
              If  set,  resources,  local jobs and attachments will be output to screen, this generates a lot of
              text and is mainly for debugging.  (CLI only)

CHECKBOX/PLAINBOX LAUNCHERS TUTORIAL

       This document provides an explanation of why launchers are necessary, what you can achieve with them, and
       goes  over  several  examples  to  better  describe their capabilities. For a detailed reference on which
       settings are supported by launchers, and specific syntax for launcher files, look at /profiles.

   Legacy checkbox behavior control
       In the past, Checkbox's behavior was controlled by three mechanisms.

       First, the functions of checkbox could be augmented by adding  plugins.   For  example,  the  ability  to
       submit  to  certification  website  was  added  by the checkbox-certification package using a plugin. The
       plugins included by checkbox-certification and which add new behavior to base checkbox were:

          /usr/share/checkbox-certification/plugins/certify_message.py
          /usr/share/checkbox-certification/plugins/submission_info.py
          /usr/share/checkbox-certification/plugins/backup.py
          /usr/share/checkbox-certification/plugins/certify_prompt.py
          /usr/share/checkbox-certification/plugins/certify_report.py
          /usr/share/checkbox-certification/plugins/certify_schemas.py

       These added the way to prompt the user for submission-specific data, generate the xml report,  and  other
       functions.

       Next, the behaviors of the plugins could be configured or controlled using configuration files, which are
       "cascaded". A config file can include others and those can in turn include others.

       This is an example of a project-specific project-qt.ini main config file. It's the first file  read  when
       the project-specific client is launched. Some settings are abbreviated:

          [DEFAULT]
          includes = %(checkbox_oem_share)s/configs/checkbox-project-base-qt.ini %(checkbox_project_share)s/configs/checkbox-project-base.ini

          [checkbox/plugins/environment_info]
          repositories = deb http://.*\(archive\|security\).ubuntu.com/ubuntu precise-security
          routers = multiple
          server_iperf = 10.20.30.40
          sources_list = /etc/apt/sources.list
          wpa_n_psk = password
          wpa_n_ssid = access-point

          [checkbox/plugins/user_interface]
          title = My project System Testing

       Notice  the  includes  line,  this  instructs it to load the config file for checkbox-project-base-qt and
       checkbox-project-base.  Checkbox-project-base-qt  loads  the  configs  for   checkbox-certification   and
       checkbox-project.  Settings  are  cascaded  so the config options near the top override the ones near the
       bottom.

       Finally, the "binary" used to invoke checkbox is a shell script that defines where  to  find  the  things
       checkbox  needs  to  run:  you  can  define  a  share  directory,  a  specific data directory, point to a
       configuration file and define some environment variables that you may  need  during  testing.  Here's  an
       example for checkbox-project-qt:

          #!/bin/bash
          export CHECKBOX_DATA=${CHECKBOX_DATA:-~/.checkbox}
          export CHECKBOX_SHARE=${CHECKBOX_SHARE:-/usr/share/checkbox}
          export CHECKBOX_OPTIONS=${CHECKBOX_OPTIONS:---log-level=debug --log=$CHECKBOX_DATA/checkbox-project.log}
          export CHECKBOX_CERTIFICATION_SHARE=${CHECKBOX_CERTIFICATION_SHARE:-/usr/share/checkbox-certification}
          export CHECKBOX_OEM_SHARE=${CHECKBOX_PROJECT_BASE_SHARE:-/usr/share/checkbox-project-base}
          export CHECKBOX_PROJECT_SHARE=${CHECKBOX_PROJECT_SHARE:-/usr/share/checkbox-project}

          # Convenience for defining the PYTHONPATH directory.
          if [ "$CHECKBOX_SHARE" != "/usr/share/checkbox" ]; then
              export PYTHONPATH="$CHECKBOX_SHARE:$PYTHONPATH"
          fi

          python3 $CHECKBOX_SHARE/run "$@" $CHECKBOX_PROJECT_SHARE/configs/$(basename $0).ini

       Here you can see that it defines some locations and an important part is the final python3 line, where it
       will locate and use the required .ini config file we saw earlier.

       This hierarchical organization was very powerful but was also difficult to  handle,  and  also  had  some
       limitations. Part of the work we did with checkbox was to integrate all the project-specific plugins into
       checkbox trunk, this way all the core code is in one place, and the project-specific variants only supply
       jobs, whitelists, data and configuration, without adding new behavior.

   New plainbox behavior control
       Unlike checkbox, plainbox's core is monolythic, and it has no concept of plugins. This makes it easier to
       understand and work with. The plainbox core has implementations for all the functions in the old checkbox
       packages,  so  no  additions  are necessary to use features such as submission to certification or report
       generation.

       What we call plainbox is the library that implements all the functionality, as can be seen here.

       Plainbox provides tools to help test  developers  write  and  package  tests.   These  are  delivered  in
       "providers",  which  are  entities designed to encapsulate test descriptions, custom scripts for testing,
       whitelists and assorted data. They were designed to allow teams to write and deliver their  custom  tests
       without worrying too much about the underlying plainbox code.

       To get information on how to write tests and providers, see the Provider Tutorial

       However,  when  actually using these tests to verify a real system, we wanted to provide something easier
       and closer to the user experience of checkbox. We created two  clients,  checkbox-gui  and  checkbox-cli,
       which  had some hardcoded behaviors, and we also started creating other clients which were based on these
       but were purpose specific. For instance, we had a version of checkbox for SRU testing, another for server
       certification, and so on.

       But then we realized that a lot of the code was duplicated and the behaviors were common except for a few
       changes. So we came up with the  concept  of  "launchers",  which  are  somewhat  similar  to  checkbox's
       configuration files and shell script launchers.

       The idea is that checkbox-gui and checkbox-cli have some very basic behaviors, since they are the clients
       that are shipped by default with ubuntu. They can show all the available whitelists,  show  a  predefined
       welcome  message,  and  at the end will let the user see the html report and submit it to launchpad using
       their e-mail address, similar to the version of checkbox that shipped with Ubuntu.

       Instead of using complicated command line switches,  launchers  allow  you  to  configure  some  optional
       behaviors  to  customize your testing experience. A launcher contains settings, and is similar to a shell
       script, but the interpreter will be either checkbox-gui or checkbox-launcher.

       Here are a few examples of what can be done with launchers.

       As a surprise, checkbox-cli is itself a launcher:

          #!/usr/bin/env checkbox-launcher
          [welcome]
          text = Welcome to System Testing!
              Checkbox provides tests to confirm that your system is working properly.
              Once you are finished running the tests, you can view a summary report for
              your system.
              Warning: Some tests could cause your system to freeze or become
              unresponsive. Please save all your work and close all other running
              applications before beginning the testing process.

          [suite]
          whitelist_filter = ^default$
          whitelist_selection = ^default$
          skip_whitelist_selection = True

          [transport]
          submit_to = launchpad

       You can see here we customize a few options: it  shows  a  welcome  message,  automatically  selects  the
       default whitelist, and will submit to launchpad when it's finished.

       A graphical launcher example is canonical-certification-client.

          #!/usr/bin/checkbox-gui

          [welcome]
          title = "System Certification"
          text = "<p>Welcome to System Certification!</p><p></p><p>This application will
          gather information from your system. Then you will be asked manual tests to
          confirm that the system is working properly. Finally, you will be asked for
          the Secure ID of the computer to submit the information to the certification
          database.</p><p></p><p> To learn how to create or locate the Secure ID,
          please see here: <a href=\"https://certification.canonical.com\">certification.canonical.com</a></p><p></p>"

          [suite]
          whitelist_filter = "^client-(cert|selftest).*"

          [submission]
          input_type = "regex"
          input_placeholder = "Secure ID (15 or 18 characters)"
          ok_btn_text = "Submit Results"
          submit_to_hexr = "true"

          [exporter]
          xml_export_path = "/tmp/submission.xml"

          [transport]
          submit_to = "certification"

       Graphical  launchers  are a bit more complicated, but essentially it's similar, what it allows is for you
       to define some parameters to customize your testing experience.

       A very  simple  text-mode  launcher  is  canonical-hw-collection  which  just  runs  the  basic  hardware
       information tests and uploads them to a hardware database:

          [welcome]
          title = Gathering hardware information
          text = Gathering hardware information.  You may be prompted for your password.
                 This process will take approximately 30 seconds and you will be provided
                 with a URL through which you can confirm and register your hardware
                 submission.

          [suite]
          whitelist_filter = ^hwsubmit$
          whitelist_selection = ^hwsubmit$
          skip_whitelist_selection = True
          skip_test_selection = True

          [submission]
          # A bogus secure_id ensures we don't ask it
          # It can always be overridden in the .conf file.
          secure_id = 000

          [transport]
          submit_to = certification
          submit_url = https://hardware-server.example.com/

       FInally,  canonical-driver-test-suite  provides  both  a  graphical  and  a text mode launcher, which are
       functionally equivalent:

          #!/usr/bin/checkbox-gui

          [welcome]
          title = "Canonical Driver Test Suite"
          text = "<p>Welcome to the Canonical Driver Test Suite.</p>
           <p></p>
           <p>This program contains automated and manual tests to help you discover
           issues that will arise when running your device drivers on Ubuntu.</p>
           <p></p>
           <p>This application will step the user through these tests in a
           predetermined order and automatically collect both system information as
           well as test results. It will also prompt the user for input when manual
           testing is required.</p>
           <p></p>
           <p>The run time for the tests is determined by which tests you decide to
           execute. The user will have the opportunity to customize the test run to
           accommodate the driver and the amount of time available for testing.</p>
           <p></p>
           <p>To begin, simply click the Continue button below and follow the onscreen
           instructions.</p><p></p>"

          [suite]
          whitelist_filter = "^ihv-.*"

          [submission]
          ok_btn_text = "Save Results"
          input_type = "none"

          [exporter]
          xml_export_path = ""

          [transport]
          submit_to = "local"

       Text mode:

          #!/usr/bin/env checkbox-launcher
          [welcome]
          text = Welcome to the Canonical Driver Test Suite
              This program contains automated and manual tests to help you discover
              issues that will arise when running your device drivers on Ubuntu.
              This application will step the user through these tests in a
              predetermined order and automatically collect both system information as
              well as test results. It will also prompt the user for input when manual
              testing is required.
              The run time for the tests is determined by which tests you decide to
              execute. The user will have the opportunity to customize the test run to
              accommodate the driver and the amount of time available for testing.
              To begin, simply click the Continue button below and follow the onscreen
              instructions.

          [suite]
          # Whitelist(s) displayed in the suite selection screen
          whitelist_filter = ^ihv-.*
          # Whitelist_selection is mandatory so we set it to a bogus value so
          # no whitelists are preselected.
          whitelist_selection = bogus

CHECKBOX RELEASE PROCESS

       This page describes the necessary steps for releasing versions of Checkbox and Checkbox Certification  to
       the stable PPA belonging to the Hardware Certification team, on a regular basis. Throughout this document
       the term 'Checkbox' is used as a catch-all term to cover all versions of Checkbox owned by  the  Hardware
       Certification team, currently Checkbox itself and the Checkbox Certification extensions.

   Overview
       Currently  the  process runs on a bi-weekly cadence, with a new release of Checkbox every two weeks. This
       covers ten working days, and the tasks carried out on each day or group of days is described below:

       • Days 1-4: Time allowed for new changes to be introduced into trunk.

       • Day 5: Changes are merged  from  the  trunk  of  lp:checkbox  and  lp:checkbox-certification  to  their
         respective  release  branches.   The  changelogs  for  both  are bumped at this point and revisions are
         tagged.  At this stage it may also be necessary to copy the package 'fwts' from  the  FWTS  Stable  PPA
         <https://launchpad.net/~firmware-testing-team/+archive/ppa-fwts-stable> to the Checkbox Release Testing
         PPA <https://launchpad.net/~checkbox-dev/+archive/testing>.

       • Days 6-9: Testing is performed by the release manager  for  the  Hardware  Certification  team,  and  a
         representative of the CE QA team (the main customer for Checkbox within Canonical)

       • Day  9:  A  release meeting is held between the release manager for the Hardware Certification team and
         the representative of the CE QA team.  Potential issues with the release are identified and plans  made
         to address them.

       • Day 10: The tested version of Checkbox is copied to the stable PPA.

   Launchpad Branches
       The  release process requires separate branches in Launchpad containing a semi-frozen version of the code
       that was in trunk on day 5 of the process. This is so that development  can  continue  on  trunk  without
       jeopardising  the  stability  of  the  to-be  released  version of Checkbox. The relationship between all
       branches involved in the process is as shown below:

       • lp:checkbox/release <- lp:checkboxlp:checkbox-certification/release <- lp:checkbox-certificationlp:~checkbox-dev/checkbox/checkbox-packaging-release <- lp:~checkbox-dev/checkbox/checkbox-packaging

   Auditing milestoned bugs
       Prior to creating the release candidate the release manager should review the list of bugs milestoned for
       the      next      release     of     Checkbox.     They     should     visit     checkbox     milestones
       <https://launchpad.net/checkbox/+milestonesmilestones> and locate the milestone dated  with  the  release
       date.

       • For  bugs  that are set to In Progress with a branch associated - liase with the branch owner to see if
         the merge can be completed before the deadline.

       • For bugs that are in any other non-closed status (except Fix  Commited)  -  re-milestone  them  to  the
         following milestone.

   Cutting the release
       In order to cut the release, we have to merge the changes from trunk into the release branch, commit them
       with a suitable message and update the changelog in trunk so that future changes  go  under  the  correct
       version. For each combination of branches shown above, do the following (the example uses lp:checkbox and
       lp:checkbox/release):

          bzr branch lp:checkbox/release checkbox-release
          bzr branch lp:checkbox checkbox-trunk
          cd checkbox-release
          current_stable=`head -n1 $(find . -name 'changelog') | grep -oP '(?<=\().*(?=\))'`
          bzr merge lp:checkbox

       at this point if no changes (other than one to debian/changelog) get merged in then we do not  perform  a
       release  of the package in question. In practice this often happens with checkbox-certification but never
       with checkbox:

          bzr commit -m "Merged in changes from rev$(bzr revno -r tag:$current_stable lp:checkbox) to rev$(bzr revno lp:checkbox) from lp:checkbox"
          bzr push lp:checkbox/release
          cd `find . -name 'debian'`; cd ..
          bzr tag `head -n1 debian/changelog | grep -oP '(?<=\().*(?=\))'`
          dch -r (save modified changelog)
          dch -i -U 'Incremented changelog'
          debcommit
          bzr push lp:checkbox

       The last step in the process is to perform a build of the packages in the  ppa:checkbox-dev/testing  PPA.
       To  do  this  we  need  to  go to the recipe pages for the checkbox and/or checkbox-certification release
       branches.

          • checkbox-testing recipe <https://code.launchpad.net/~checkbox-dev/+recipe/checkbox-testing>

          • checkbox-certification-testing  recipe   <https://code.launchpad.net/~checkbox-dev/+recipe/checkbox-
            certification-testing>

       The Build Now option should be available on the page. Click it to start a build.

   Copying Firmware Test Suite to the Testing PPA
       The  Firmware  Test  Suite  tool is a test tool for system firmware that is naturally heavily utilised by
       Checkbox. To make sure the latest version which contains fixes and new tests/features needed by  Checkbox
       is available and also doesn't break anything in Checkbox, we need to release it alongside Checkbox. After
       cutting the release if the Firmware Testing team have notified that a new version is available  and  that
       this  version should be used for certification, we need to copy it to the Testing PPA. To do this we need
       to go to the Copy packages view of the Firmware Test Suite (Stable) PPA <https://launchpad.net/~firmware-
       testing-team/+archive/ppa-fwts-stable/+copy-packages>  and  select  the  'fwts' packages for all releases
       back  to  Precise.  We   need   to   set   the   'Destination   PPA'   as   'Checkbox   Release   Testing
       [~checkbox-dev/testing]'  and  the  'Copy  options'  field  to 'Copy existing binaries', then click 'Copy
       packages'. This step then needs to be repeated but set the 'Destination PPA' field to 'PPA  for  Checkbox
       Developers [~checkbox-dev/ppa]'.

   Next Release of Checkbox e-mail
       So  that  everyone  has the opportunity to perform whatever testing is required in a timely manner, after
       the PPA builds have been completed an email should be sent to the following mailing lists:

       • hardware-certification-team@lists.canonical.com <hardware-certification-team@lists.canonical.com>

       • commercial-engineering@lists.canonical.com <commercial-engineering@lists.canonical.com>

       The content is typically something like this:

          Subject: Next Release of Checkbox (18/11/2013)

          Hi,

          The next release of Checkbox is available in the
          https://code.launchpad.net/~checkbox-dev/+archive/testing PPA.
          Please test it at your convenience. Checkbox is based on revision 2484 of
          lp:checkbox and Checkbox Certification is based on revision 586 of
          lp:checkbox-certification.

          Thanks,

       If one or the other of Checkbox and Checkbox Certification have not been updated then there is no need to
       mention that package

   Testing the release
       Now that the release has been cut, testing should take place prior to the release meeting. From the point
       of view of the  certification  team,  what  needs  to  be  tested  is  checkbox-certification-client  and
       checkbox-certification-server which form the basis for CE QAs OEM specific versions of Checkbox. Checkbox
       certification server is tested in the CI loop Checkbox certification client needs to be tested manually.

   Release Meeting
       On the Thursday before the  release  is  made,  a  meeting  is  held  between  a  representative  of  the
       Certification  team  and  a  representative of the Commercial Engineering QA team. The meeting is held at
       7:30          UTC          as          shown          in          this          calendar           invite
       <https://www.google.com/calendar/hosted/canonical.com/event?action=TEMPLATE&tmeid=Y3QxcWVla3ViMTRvMXByOHZlOTFvc283Y2NfMjAxMzA4MjlUMDczMDAwWiBicmVuZGFuLmRvbmVnYW5AY2Fub25pY2FsLmNvbQ&tmsrc=brendan.donegan%40canonical.com>.
       An agenda for the meeting is included in the invite.

   Publishing the release
       To publish the release we simply need to copy a number of packages from the Checkbox Release Testing  PPA
       <https://launchpad.net/~checkbox-dev/+archive/testing>   to   the   Hardware   Certification  Public  PPA
       <https://launchpad.net/~hardware-certification/+archive/public>. To do this we go to  the  Copy  packages
       view  of  the  Checkbox  Release Testing PPA <https://launchpad.net/~checkbox-dev/+archive/testing/+copy-
       packages> and select all versions of the following list of  packages:  checkbox,  checkbox-certification,
       fwts.  Make  sure  that  the  'Destination  PPA'  field  is set to 'Public PPA for Hardware Certification
       [~hardware-certification/public]' and that the 'Copy options' field is set to 'Copy  existing  binaries',
       then click 'Copy Packages'.

       After  that  is  done  an announcement email should be sent to commercial-engineering@lists.canonical.com
       <commercial-engineering@lists.canonical.com>.  A template for the announcement in included below:

          Hi,

          A new release of checkbox has been uploaded to the Hardware
          Certification Public PPA
          (https://launchpad.net/~hardware-certification/+archive/public). The
          release is based on revision 2294 of lp:checkbox

          Thanks,

       Please attach the most recent part of the changelog as release notes

       • genindex

       • modindex

       • search

AUTHOR

       Zygmunt Krynicki

       2013-2014, Zygmunt Krynicki