Ubuntu Wiki

Introduction

Checkbox (https://launchpad.net/checkbox) is a test runner for Ubuntu. It aims to provide a common framework to run all types of tests, from hardware tests, to command line tests, unit tests or desktop tests and send their results to Launchpad, automatically.

General Configuration

Checkbox general configuration is stored at /etc/checkbox.d. Each way of calling checkbox (GUI, cli) has its own configuration file.

checkbox.ini

This is the general config file. Properties listed here will be inherited by all checkbox flavours.

Example:

[DEFAULT]
version = 0.9.1
plugins = checkbox/plugins

[checkbox/plugins]
modules = %(checkbox_share)s/plugins/*.py

Explanation of options:

• [DEFAULT] - This section covers all general configuration properties.

• version - The version of checkbox to use the configuration with.

• [checkbox/plugins] - This section controls the checkbox plugin system.

• modules - The path to the checkbox plugin executable scripts.

• checkbox_share - Where the checkbox core files are located. By default: /usr/share/checkbox

Jobs

What is a job?

A job is a task or unit of work that is used to define many different things including test suites and test cases. All jobs are composed of a list of fields that have a special meaning.

For more information about writing jobs, please refer to the following sections:

• Writing tests How-To

• Walkthrough

What fields may a job include?

A job can include any of the following fields:

• name: Unique identifier for a job
• plugin: Takes care of the interpretation of the job (describes the type of job: maunal, automated, resource, etc)
• depends: The job will only run if it's dependency job passes. If JobA depends on JobB, and JobB fails, then JobA will be skipped.

• requires: Set of requirements that must be satisfied for the job to be executed. Typical requirements are of the kind some package must be installed or some hardware must be detected.

• timeout: How much time should checkbox wait before cancelling job execution
• command: A string that must be executed using the same commands as in a shell

• user: Don't execute command with the same user that launched checkbox, but with a different user (usually root)

• description: Human-readable description of the job purpose

Are there any mandatory fields?

Yes, name and plugin must always be present. Depending on the plugin, description and command may also be required. However, it's advised to use description in every job (even when not needed) for documentation purposes.

Where are jobs stored?

In source code you'll always find jobs under jobs directory. In an installed package, such as checkbox, the jobs are under /usr/share/checkbox/jobs directory.

Which format is used for jobs?

Jobs are stored using rfc822 format. Despite this might not seem a very convenient format, it's really useful because it supported by the translations tools and this makes possible to translate jobs easily.

What's the difference between .txt.in and .txt?

On one hand, in source code you'll find that jobs use the .txt.in extension. This is because in they are translations templates that contain extra information information about which strings are expected to be translated.

On the other hand, in the package you'll find that jobs use the .txt extension. This is because they are just job description files and the information for translations has been processed and stored in special files.

Why _description instead of description?

In rfc822 files, the underscore prefix is used to mark a field as translatable. Hence all .txt.in files use this prefix to mark description as a translatable string.

Is there an easy way to write job files?

Checkbox Editor is a tool that lets you write jobs easily without having to take care of the formatting required. This tool is available in the OEM Community PPA and it already includes a complete manual.

Plugins

What is a plugin?

A plugin, in the checkbox vocabulary, is a script that takes care of the interpretation of a job, that's to say, it implements the policy that is going to be applied to that job.

What plugins are available?

The list of currently available plugins is:

• local: Include local test suites that have been written for checkbox. Local jobs are run first and are usually used to create dynamic job descriptions (e.g. create a storage test for every detected hard drive) or for other pre-testing needs such as generating test suites (e.g. parsing the name.txt file and loading all the job descriptions contained in the file).

• remote: Integrate remote third party test suites that have been written for another framework (i.e. LTP, mago, ...) *NOTE that remote jobs only pass or fail based on the return code of the external framework. So if running LTP and 10 of 50 tests fail, but LTP itself returns a 0 exit status, the remote job will be marked as successful.

• manual: manual test cases, or those which the user must perform and action and evaluate the results. For example: "Plug in headphones and select the Test button. Did you hear the sound clearly?"

• shell: Automated test cases, those which are run automatically by checkbox and do not require or ask for any user action at all.

• attachment: Add attachment to report, that is, important information that must be kept for reference

• resource: Gather hardware resource information

What plugins do I really need to write tests?

For test suites: local. For test cases: manual and shell (for non-interactive tests).

Implementation

A plugin is simply a Python module which registers for events. A minimal plugin would look like:

class Cr3Plugin(Plugin):
    def register(self, manager):
       super(Cr3Plugin, self).register(manager)
       self._manager.reactor.call_on("report", self.report)
       
    def report(self):
       self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

factory = Cr3Plugin

A plugin should inherit from the Plugin class and it should register to be called on particular events. In this example it is registered to call the report method upon the "report" event.

self._manager.reactor.call_on("report", self.report)

The report method itself fires the "report-cr3" and anyone who has registered for this particular event will be called.

self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

The plugin should provide a global factory variable which will be the hook for instantiating your plugin.

Resources

What is a resource?

A resource is a specialized job that uses the plugin of the same name. In addition to this, it also needs a command that, when executed, prints to the standard output pieces of information in rfc822 format that the resource plugin is able to parse.

What are resources used for?

Resources are used to gather information that is suitable to be added as requirements in other jobs.

For example, if a job requires the totem package to be executed, then the following lines can be added to the job description:

requires:
 package.name == 'totem'

and checkbox will make sure that, in the information gathered by the package resource, there's a field called name that matches the string totem.

Where can I find some examples of resource jobs?

Some examples of resource jobs can be found in the /usr/share/checkbox/jobs/resource.txt file. In particular, the package resource shown in the previous section:

name: package
plugin: resource
command: package_resource | filter_templates -w 'desired=Install' -w 'status=Installed'

As it can be guessed from the command line the information that is gathered in this case is the list of installed packages. If the command is executed, it will print blocks of rfc822 data with some fields regarding the package being printed. In particular, for the totem package in the previous section example, the output is:

status: Installed
desired: Install
version: 2.30.2-0ubuntu1
name: totem
description: A simple media player for the GNOME desktop based on GStreamer

where the name field indeed contains the totem string as was required in the previous section example.

Tips And Tricks

[/TipsAndTricks]