Checkbox

Differences between revisions 22 and 23
Revision 22 as of 2009-04-28 19:50:51
Size: 8707
Editor: pool-173-76-105-89
Comment:
Revision 23 as of 2009-06-09 09:50:06
Size: 8821
Editor: 157
Comment:
Deletions are marked like this. Additions are marked like this.
Line 59: Line 59:

For more information about writing test cases, please refer to the [[/WritingTestsHowTo|Writing tests How-To]]

Introduction

Checkbox (https://launchpad.net/checkbox) is a test runner for Ubuntu. It aims to provide a common framework to run all types of tests, from hardware tests, to command line tests, unit tests or desktop tests and send their results to Launchpad, automatically.

General Configuration

Checkbox general configuration is stored at /etc/checkbox.d. Each way of calling checkbox (GUI, cli) has its own configuration file.

checkbox.ini

This is the general config file. Properties listed here will be inherited by all checkbox flavours.

Example:

[DEFAULT]
version = 0.7
plugins = checkbox/plugins
registries = checkbox/registries

[checkbox/plugins]
blacklist = backend_manager
modules = %(checkbox_share)s/plugins/*.py

[checkbox/registries]
modules = %(checkbox_share)s/registries/*.py

Explanation of options:

  • [DEFAULT] - This section covers all general configuration properties.

    • version - The version of checkbox to use the configuration with.

    • plugins - Folder where checkbox plugins are stored.

    • registries - Folder where checkbox registries are stored.

  • [checkbox/plugins] - This section controls the checkbox plugin system.

    • blacklist - Plugins to be ignored when running checkbox.

    • modules - The path to the checkbox plugin executable scripts.

  • [checkbox/registries] - This section controls the checkbox registry system.

    • modules - The path to the checkbox registry executable scripts.

Tests

Test are defined in plain text files, that can be stored anywhere.

The files consist of some required fields and some optional fields.

Let's look at one example:

name: audio
plugin: manual
categories: laptop, desktop
requires: alsa.type == 'control'
command: audio_playback $data_path/audio_playback.wav
description: Testing detected sound card: . $(audio_playback --view) . Did you hear a sound?

For more information about writing test cases, please refer to the Writing tests How-To

Tests fields

Required fields

Field

Explanation

name

Unique name for a test

plugin

Plugin name to handle this test. Checkbox provides "manual" and "shell" by default

description

A description on what the test does. In the case of manual tests, this is the actual question asked to the enduser. In the case of automated tests, this is strictly informational.

Optional fields

Field

Explanation

architectures

List of architectures for which this test is relevant: amd64, i386, powerpc and/or sparc

categories

List of categories for which this test is relevant: desktop, laptop and/or server

command

Command to run for the test. This is not required for manual tests, but it can be used, anyway.

depends

List of names of tests on which this test depends. So, if the other test fails, this test will be skipped.

requires

Registry expressions which are requirements for this test: 'input.mouse' in info.capabilities

timeout

Timeout for running the command.

optional

Boolean expression set to True if this test is optional or False if this test is required.

Special fields

  • $output: This will be changed by the output of the command. Example:

command: date
description:
 was uds good for you on $output?

In this case, the description will show the current date. Clicking on "Test again" will re-render the description.

Test Suites

Tests can be grouped by suites. A suite is basically a text file that contains one or more tests. Suites are available under the directory: /usr/share/checkbox/suites

All the files under that directory are automatically parsed for tests.

Test suites are parsed by the suites_info plugin (check the Plugin section for more information about plugins)

Environment variables

Apart from the two options explained above, this is the place to define the environment variables that we want for our tests. From the example above this would be:

data_path = %(checkbox_directory)s/data

Creating Tests and Suites

For a more detailed walkthrough of creating checkbox tests and suites, see this walkthrough.

Testing your suites in a bzr checkout of checkbox

If you are adding new suites to checkbox, you can simply test them from within the checkout, you can test directly from there:

./bin/checkbox-gtk --config="checkbox/plugins/suites_info/whitelist=mysuite.txt.in"

Easiest way to run your test with Checkbox

If you don't need any particular parsing of the results and you only want to run your test with Checkbox all you need to do is:

  • Prepare one or more scripts to perform the test.
  • Place the script(s) in /usr/share/checkbox/scripts.
  • Add the test to an existing suite (or create a new suite) in /usr/share/checkbox/suites.
  • Run checkbox.

Test results

Test results are sent automatically to Launchpad if an internet connection is available. Apart from sending them, they are stored temporarily at /var/cache/checkbox. This is the default folder, but it can be changed with the environment variable CHECKBOX_DATA.

To change the results folder in run time just start the application with:

CHECKBOX_DATA=/tmp sudo -E checkbox-gtk

Plugins and Registries

Plugins and registries control how checkbox functions, executes scripts, and gathers information. While they are fully configurable and extensible, most tasks can be accomplished with the existing plugins and registries.

Plugins

A large number of plugins are provided by default, and can be seen in /usr/share/checkbox/plugins. When writing tests, the most common plugins used will be manual_prompt (for manual tests) and shell_prompt (for shell or non-interactive tests).

Description

Plugins change Checkbox default behaviour. In fact, the checkbox default behaviours are plugins themselves.

Default Plugins

Implementation

A plugin is simply a Python module which registers for events. A minimal plugin would look like:

class Cr3Plugin(Plugin):
    def register(self, manager):
       super(Cr3Plugin, self).register(manager)
       self._manager.reactor.call_on("report", self.report)
       
    def report(self):
       self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

factory = Cr3Plugin

A plugin should inherit from the Plugin class and it should register to be called on particular events. In this example it is registered to call the report method upon the "report" event.

self._manager.reactor.call_on("report", self.report)

The report method itself fires the "report-cr3" and anyone who has registered for this particular event will be called.

self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

The plugin should provide a global factory variable which will be the hook for instantiating your plugin.

Registries

Description

Registries provide a consistent way of accessing hierarchical information in the system. They are meant to expose a single rooted data structure which can be accessed consistently regardless of where the information might be stored.

Contents

The existing registries give fast access to a large amount of system information. This includes:

  • CPU information
  • PCI and USB bus information
  • HAL information
  • Installed package status (dpkg)
  • GConf contents
  • Kernel modules
  • Drive mount information
  • Extended system information (through sysctl)

Registries can be queried via a simple text search within a suite.

Implementation

One of the motivations for providing this kind of data structure is to be able to define the requires: field in tests in a consistent way. Another is to provide plugins with a convenient way to obtain and query information about the system.

A registry is simply a Python module that extends the Registry class. A minimal registry would look like:

class Cr3Registry(Registry):
   
   def items(self):
       return (("foo": 1), ("bar": 2))

factory = Cr3Registry

The new Registry simply needs to define an items method and then the RegistryManager handles the rest.

Tips And Tricks

[/TipsAndTricks]

Testing/Automation/Checkbox (last edited 2021-10-20 11:49:54 by sylvain-pineau)