Checkbox

Differences between revisions 5 and 24 (spanning 19 versions)
⇤ ← Revision 5 as of 2008-10-20 21:49:40 →
Size: 3731
Editor: modemcable178
Comment:
← Revision 24 as of 2010-04-30 08:58:22 → ⇥
Size: 5964
Editor: 43
Comment:
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
Checkbox (https://launchpad.net/checkbox) is a test runner for Ubuntu. It aims is to provide a common framework to run all types of tests, from hardware tests, to command line tests, unit tests or desktop tests and send their results to Launchpad, automatically. Checkbox (https://launchpad.net/checkbox) is a test runner for Ubuntu. It aims to provide a common framework to run all types of tests, from hardware tests, to command line tests, unit tests or desktop tests and send their results to Launchpad, automatically.
Line 14: Line 14:
This is the general config file. It will be inherit by all the checkbox flavours. This is the general config file. Properties listed here will be inherited by all checkbox flavours.
Line 16: Line 16:
 * [DEFAULT]. Under this section they are covered all general configuration properties.
  * version. The version of checkbox to use the configuration with.
  * plugins. Folder where checkbox plugins are stored.
  * registries. Folder where checkbox registries are stored.		 Example:
{{{
[DEFAULT]
version = 0.9.1
plugins = checkbox/plugins
Line 21: Line 22:
== Tests == [checkbox/plugins]
modules = %(checkbox_share)s/plugins/*.py
}}}
Line 23: Line 26:
Test are defined in plain text files, that can be stored anywhere. Explanation of options:
Line 25: Line 28:
They consists in some required fields and some optional fields.  * '''[DEFAULT]''' - This section covers all general configuration properties.
 * ''version'' - The version of checkbox to use the configuration with.
 * '''[checkbox/plugins]''' - This section controls the checkbox plugin system.
 * ''modules'' - The path to the checkbox plugin executable scripts.
Line 27: Line 33:
Let's look at one example: == Jobs ==
=== What is a job? ===
A job is a task or unit of work that is used to define many different things including test suites and test cases. All jobs are composed of a list of '''fields''' that have a special meaning.

For more information about writing jobs, please refer to the following sections:
  * [[/WritingTestsHowTo|Writing tests How-To]]
  * [[/WalkThrough|Walkthrough]]

=== What fields may a job include? ===
A job can include any of the following fields:
 * name: Unique identifier for a job
 * plugin: Takes care of the interpretation of the job
 * depends: The the job cannot be executed, at least until another one has been executed successfully
 * requires: Set of requirements that must be satisfied for the job to be executed. Typical requirements are of the kind ''some package must be installed'' or ''some hardware must be detected''.
 * timeout: How much time should checkbox wait before cancelling job execution
 * command: A string that must be executed using the same commands as in a shell
 * user: Don't execute command with the same user that launched checkbox, but with a different user (usually ''root'')
 * description: Human-readable description of the job purpose

=== Are there any mandatory fields? ===
Yes, '''name''' and '''plugin''' must always be present. Depending on the ''plugin'', '''description''' and '''command''' may also be required. However, it's advised to use ''description'' in every job (even when not needed) for documentation purposes.

=== Where are jobs stored? ===
In source code you'll always find jobs under '''jobs''' directory.
In an installed package, such as ''checkbox'', the jobs are under '''/usr/share/checkbox/jobs''' directory.

=== Which format is used for jobs? ===
Jobs are stored using rfc822 format. Despite this might not seem a very convenient format, it's really useful because it supported by the translations tools and this makes possible to translate jobs easily.

=== What's the difference between .txt.in and .txt? ===
On one hand, in source code you'll find that jobs use the ''.txt.in'' extension. This is because in they are translations templates that contain extra information information about which strings are expected to be translated.

On the other hand, in the package you'll find that jobs use the ''.txt'' extension. This is because they are just job description files and the information for translations has been processed and stored in special files.

=== Why _description instead of description? ===
In rfc822 files, the underscore prefix is used to mark a field as translatable. Hence all ''.txt.in'' files use this prefix to mark description as a translatable string.

== Plugins ==
=== What is a plugin? ===
A plugin, in the checkbox vocabulary, is a script that takes care if the interpretation of a ''job'', that's to say, it implements the policy that it's going to be applied to that ''job''.

=== What plugins are available? ===
The list of currently available plugins is:
 * local: Include ''local'' test suites that have been written for checkbox.
 * remote: Integrate ''remote'' third party test suites that have been written for another framework (i.e. LTP, mago, ...)
 * manual: Write ''manual'' test cases
 * shell: Write a automated ''shell'' test cases
 * attachment: Add ''attachment'' to report, that is, important information that must kept for reference
 * resource: Gather hardware ''resource'' information

=== What plugins do I really need to write tests? ===
For test suites: '''local'''. For test cases: '''manual''' and '''shell''' (for non-interactive tests).

=== Implementation ===
A plugin is simply a Python module which registers for events. A minimal plugin would look like:
Line 30: Line 90:
name: audio
plugin: manual
categories: laptop, desktop
requires: alsa.type == 'control'
command: audio_playback $data_path/audio_playback.wav
description: Testing detected sound card: . $(audio_playback --view) . Did you hear a sound?		 class Cr3Plugin(Plugin):
    def register(self, manager):
       super(Cr3Plugin, self).register(manager)
       self._manager.reactor.call_on("report", self.report)
       
    def report(self):
       self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

factory = Cr3Plugin
Line 38: Line 101:
=== Tests fields === A plugin should inherit from the Plugin class and it should register to be called on particular events. In this example it is registered to call the report method upon the "report" event.
{{{
self._manager.reactor.call_on("report", self.report)
}}}
Line 40: Line 106:
==== Required fields ==== The report method itself fires the "report-cr3" and anyone who has registered for this particular event will be called.
{{{
self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")
}}}
Line 42: Line 111:
|| '''Field''' || '''Explanation''' ||
|| name || Unique name for a test ||
|| plugin || Plugin name to handle this test. Checkbox provides "manual" and "auto" by default ||
|| description || A description on what the test does. In the case of manual tests, this is the actual question asked to the enduser. In the case of automated tests, this is strictly informational. ||		 The plugin should provide a global factory variable which will be the hook for instantiating your plugin.
Line 47: Line 113:
==== Optional fields ==== === Resources ===
Line 49: Line 115:
|| '''Field''' || '''Explanation''' ||
|| architectures || List of architectures for which this test is relevant: amd64, i386, powerpc and/or sparc ||
|| categories || List of categories for which this test is relevant: desktop, laptop and/or server ||
|| command || Command to run for the test. This is not required for manual tests, but it can be used, anyway. ||
|| depends || List of names of tests on which this test depends. So, if the other test fails, this test will be skipped. ||
|| requires || Registry expressions which are requirements for this test: 'input.mouse' in info.capabilities ||
|| timeout || Timeout for running the command. ||
|| optional || Boolean expression set to True if this test is optional or False if this test is required. ||		 TBD
Line 58: Line 117:
=== Test Suites ===

These tests can be grouped by suites. A suite is basically a text file that contains one or more tests.
Suites are available under the directory: /usr/share/checkbox/suites

All the files under that directory are automatically parsed for tests.

Tests suites are parsed by the suites_info plugin (check the Plugin section for more information about plugins)

Some of the options that we can see in the configuration file (checkbox.ini) for this plugin are:

\[checkbox/plugins/suites_info\]

 * directories = %(checkbox_directory)s/suites
   The folder or folders where test suites are stored.

 * scripts_path = %(checkbox_directory)s/scripts
   The folder where runnable scripts are stored.

==== Environment variables ====

Apart from the two options explained above, this is the place to define the environment variables that we want for our tests. From the example above this would be:

data_path = %(checkbox_directory)s/data

== Plugins ==



== Registries ==

 == Tips And Tricks ==

Introduction

Checkbox (https://launchpad.net/checkbox) is a test runner for Ubuntu. It aims to provide a common framework to run all types of tests, from hardware tests, to command line tests, unit tests or desktop tests and send their results to Launchpad, automatically.

General Configuration

Checkbox general configuration is stored at /etc/checkbox.d. Each way of calling checkbox (GUI, cli) has its own configuration file.

checkbox.ini

This is the general config file. Properties listed here will be inherited by all checkbox flavours.

Example: 

[DEFAULT]
version = 0.9.1
plugins = checkbox/plugins

[checkbox/plugins]
modules = %(checkbox_share)s/plugins/*.py

Explanation of options:

  • [DEFAULT] - This section covers all general configuration properties.

  • version - The version of checkbox to use the configuration with.

  • [checkbox/plugins] - This section controls the checkbox plugin system.

  • modules - The path to the checkbox plugin executable scripts.

Jobs

What is a job?

A job is a task or unit of work that is used to define many different things including test suites and test cases. All jobs are composed of a list of fields that have a special meaning.

For more information about writing jobs, please refer to the following sections:

What fields may a job include?

A job can include any of the following fields:

  • name: Unique identifier for a job
  • plugin: Takes care of the interpretation of the job
  • depends: The the job cannot be executed, at least until another one has been executed successfully

  • requires: Set of requirements that must be satisfied for the job to be executed. Typical requirements are of the kind some package must be installed or some hardware must be detected.

  • timeout: How much time should checkbox wait before cancelling job execution
  • command: A string that must be executed using the same commands as in a shell

  • user: Don't execute command with the same user that launched checkbox, but with a different user (usually root)

  • description: Human-readable description of the job purpose

Are there any mandatory fields?

Yes, name and plugin must always be present. Depending on the plugin, description and command may also be required. However, it's advised to use description in every job (even when not needed) for documentation purposes.

Where are jobs stored?

In source code you'll always find jobs under jobs directory. In an installed package, such as checkbox, the jobs are under /usr/share/checkbox/jobs directory.

Which format is used for jobs?

Jobs are stored using rfc822 format. Despite this might not seem a very convenient format, it's really useful because it supported by the translations tools and this makes possible to translate jobs easily.

What's the difference between .txt.in and .txt?

On one hand, in source code you'll find that jobs use the .txt.in extension. This is because in they are translations templates that contain extra information information about which strings are expected to be translated.

On the other hand, in the package you'll find that jobs use the .txt extension. This is because they are just job description files and the information for translations has been processed and stored in special files.

Why _description instead of description?

In rfc822 files, the underscore prefix is used to mark a field as translatable. Hence all .txt.in files use this prefix to mark description as a translatable string.

Plugins

What is a plugin?

A plugin, in the checkbox vocabulary, is a script that takes care if the interpretation of a job, that's to say, it implements the policy that it's going to be applied to that job.

What plugins are available?

The list of currently available plugins is:

  • local: Include local test suites that have been written for checkbox.

  • remote: Integrate remote third party test suites that have been written for another framework (i.e. LTP, mago, ...)

  • manual: Write manual test cases

  • shell: Write a automated shell test cases

  • attachment: Add attachment to report, that is, important information that must kept for reference

  • resource: Gather hardware resource information

What plugins do I really need to write tests?

For test suites: local. For test cases: manual and shell (for non-interactive tests).

Implementation

A plugin is simply a Python module which registers for events. A minimal plugin would look like: 

class Cr3Plugin(Plugin):
    def register(self, manager):
       super(Cr3Plugin, self).register(manager)
       self._manager.reactor.call_on("report", self.report)
       
    def report(self):
       self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

factory = Cr3Plugin

A plugin should inherit from the Plugin class and it should register to be called on particular events. In this example it is registered to call the report method upon the "report" event. 

self._manager.reactor.call_on("report", self.report)

The report method itself fires the "report-cr3" and anyone who has registered for this particular event will be called. 

self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

The plugin should provide a global factory variable which will be the hook for instantiating your plugin.

Resources

TBD

Tips And Tricks

[/TipsAndTricks]

Testing/Automation/Checkbox (last edited 2021-10-20 11:49:54 by sylvain-pineau)