Ubuntu Wiki

Introduction

Checkbox (https://launchpad.net/checkbox) is a test runner for Ubuntu. It aims to provide a common framework to run all types of tests, from hardware tests, to command line tests, unit tests or desktop tests and send their results to Launchpad, automatically.

General Configuration

Checkbox general configuration is stored at /etc/checkbox.d. Each way of calling checkbox (GUI, cli) has its own configuration file.

checkbox.ini

This is the general config file. Properties listed here will be inherited by all checkbox flavours.

Example:

[DEFAULT]
version = 0.9.1
plugins = checkbox/plugins

[checkbox/plugins]
modules = %(checkbox_share)s/plugins/*.py

Explanation of options:

• [DEFAULT] - This section covers all general configuration properties.

• version - The version of checkbox to use the configuration with.

• [checkbox/plugins] - This section controls the checkbox plugin system.

• modules - The path to the checkbox plugin executable scripts.

Jobs

What is a job?

A job is a task or unit of work that is used to define many different things including test suites and test cases. All jobs are composed of a list of fields that have a special meaning.

For more information about writing jobs, please refer to the following sections:

• Writing tests How-To

• Walkthrough

What fields may a job include?

A job can include any of the following fields:

• name: Unique identifier for a job
• plugin: Takes care of the interpretation of the job
• depends: The the job cannot be executed, at least until another one has been executed successfully

• requires: Set of requirements that must be satisfied for the job to be executed. Typical requirements are of the kind some package must be installed or some hardware must be detected.

• timeout: How much time should checkbox wait before cancelling job execution
• command: A string that must be executed using the same commands as in a shell

• user: Don't execute command with the same user that launched checkbox, but with a different user (usually root)

• description: Human-readable description of the job purpose

Are there any mandatory fields?

Yes, name and plugin must always be present. Depending on the plugin, description and command may also be required. However, it's advised to use description in every job (even when not needed) for documentation purposes.

Where are jobs stored?

In source code you'll always find jobs under jobs directory. In an installed package, such as checkbox, the jobs are under /usr/share/checkbox/jobs directory.

Which format is used for jobs?

Jobs are stored using rfc822 format. Despite this might not seem a very convenient format, it's really useful because it supported by the translations tools and this makes possible to translate jobs easily.

What's the difference between .txt.in and .txt?

On one hand, in source code you'll find that jobs use the .txt.in extension. This is because in they are translations templates that contain extra information information about which strings are expected to be translated.

On the other hand, in the package you'll find that jobs use the .txt extension. This is because they are just job description files and the information for translations has been processed and stored in special files.

Why _description instead of description?

In rfc822 files, the underscore prefix is used to mark a field as translatable. Hence all .txt.in files use this prefix to mark description as a translatable string.

Plugins

What is a plugin?

A plugin, in the checkbox vocabulary, is a script that takes care if the interpretation of a job, that's to say, it implements the policy that it's going to be applied to that job.

What plugins are available?

The list of currently available plugins is:

• local: Include local test suites that have been written for checkbox.

• remote: Integrate remote third party test suites that have been written for another framework (i.e. LTP, mago, ...)

• manual: Write manual test cases

• shell: Write a automated shell test cases

• attachment: Add attachment to report, that is, important information that must kept for reference

• resource: Gather hardware resource information

What plugins do I really need to write tests?

For test suites: local. For test cases: manual and shell (for non-interactive tests).

Implementation

A plugin is simply a Python module which registers for events. A minimal plugin would look like:

class Cr3Plugin(Plugin):
    def register(self, manager):
       super(Cr3Plugin, self).register(manager)
       self._manager.reactor.call_on("report", self.report)
       
    def report(self):
       self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

factory = Cr3Plugin

A plugin should inherit from the Plugin class and it should register to be called on particular events. In this example it is registered to call the report method upon the "report" event.

self._manager.reactor.call_on("report", self.report)

The report method itself fires the "report-cr3" and anyone who has registered for this particular event will be called.

self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

The plugin should provide a global factory variable which will be the hook for instantiating your plugin.

Resources

TBD

Tips And Tricks

[/TipsAndTricks]