Checkbox

Differences between revisions 23 and 24
Revision 23 as of 2009-06-09 09:50:06
Size: 8821
Editor: 157
Comment:
Revision 24 as of 2010-04-30 08:58:22
Size: 5964
Editor: 43
Comment:
Deletions are marked like this. Additions are marked like this.
Line 19: Line 19:
version = 0.7 version = 0.9.1
Line 21: Line 21:
registries = checkbox/registries
Line 24: Line 23:
blacklist = backend_manager
Line 26: Line 24:

[checkbox/registries]
modules = %(checkbox_share)s/registries/*.py
Line 34: Line 29:
  * ''version'' - The version of checkbox to use the configuration with.
  * ''plugins'' - Folder where checkbox [[#Plugins|plugins]] are stored.
  * ''registries'' - Folder where checkbox [[#Registries|registries]] are stored.
 * ''version'' - The version of checkbox to use the configuration with.
Line 38: Line 31:
  * ''blacklist'' - Plugins to be ignored when running checkbox.
  * ''modules'' - The path to the checkbox plugin executable scripts.
 * '''[checkbox/registries]''' - This section controls the checkbox registry system.
  * ''modules'' - The path to the checkbox registry executable scripts.
 * ''modules'' - The path to the checkbox plugin executable scripts.
Line 43: Line 33:
== Tests == == Jobs ==
=== What is a job? ===
A job is a task or unit of work that is used to define many different things including test suites and test cases. All jobs are composed of a list of '''fields''' that have a special meaning.
Line 45: Line 37:
Test are defined in plain text files, that can be stored anywhere. For more information about writing jobs, please refer to the following sections:
  * [[/WritingTestsHowTo|Writing tests How-To]]
  * [[/WalkThrough|Walkthrough]]
Line 47: Line 41:
The files consist of some required fields and some optional fields. === What fields may a job include? ===
A job can include any of the following fields:
 * name: Unique identifier for a job
 * plugin: Takes care of the interpretation of the job
 * depends: The the job cannot be executed, at least until another one has been executed successfully
 * requires: Set of requirements that must be satisfied for the job to be executed. Typical requirements are of the kind ''some package must be installed'' or ''some hardware must be detected''.
 * timeout: How much time should checkbox wait before cancelling job execution
 * command: A string that must be executed using the same commands as in a shell
 * user: Don't execute command with the same user that launched checkbox, but with a different user (usually ''root'')
 * description: Human-readable description of the job purpose
Line 49: Line 52:
Let's look at one example: === Are there any mandatory fields? ===
Yes, '''name''' and '''plugin''' must always be present. Depending on the ''plugin'', '''description''' and '''command''' may also be required. However, it's advised to use ''description'' in every job (even when not needed) for documentation purposes.
Line 51: Line 55:
{{{
name: audio
plugin: manual
categories: laptop, desktop
requires: alsa.type == 'control'
command: audio_playback $data_path/audio_playback.wav
description: Testing detected sound card: . $(audio_playback --view) . Did you hear a sound?
}}}
=== Where are jobs stored? ===
In source code you'll always find jobs under '''jobs''' directory.
In an installed package, such as ''checkbox'', the jobs are under '''/usr/share/checkbox/jobs''' directory.
Line 60: Line 59:
For more information about writing test cases, please refer to the [[/WritingTestsHowTo|Writing tests How-To]] === Which format is used for jobs? ===
Jobs are stored using rfc822 format. Despite this might not seem a very convenient format, it's really useful because it supported by the translations tools and this makes possible to translate jobs easily.
Line 62: Line 62:
=== Tests fields === === What's the difference between .txt.in and .txt? ===
On one hand, in source code you'll find that jobs use the ''.txt.in'' extension. This is because in they are translations templates that contain extra information information about which strings are expected to be translated.
Line 64: Line 65:
==== Required fields ==== On the other hand, in the package you'll find that jobs use the ''.txt'' extension. This is because they are just job description files and the information for translations has been processed and stored in special files.
Line 66: Line 67:
|| '''Field''' || '''Explanation''' ||
|| name || Unique name for a test ||
|| plugin || Plugin name to handle this test. Checkbox provides "manual" and "shell" by default ||
|| description || A description on what the test does. In the case of manual tests, this is the actual question asked to the enduser. In the case of automated tests, this is strictly informational. ||
=== Why _description instead of description? ===
In rfc822 files, the underscore prefix is used to mark a field as translatable. Hence all ''.txt.in'' files use this prefix to mark description as a translatable string.
Line 71: Line 70:
==== Optional fields ==== == Plugins ==
=== What is a plugin? ===
A plugin, in the checkbox vocabulary, is a script that takes care if the interpretation of a ''job'', that's to say, it implements the policy that it's going to be applied to that ''job''.
Line 73: Line 74:
|| '''Field''' || '''Explanation''' ||
|| architectures || List of architectures for which this test is relevant: amd64, i386, powerpc and/or sparc ||
|| categories || List of categories for which this test is relevant: desktop, laptop and/or server ||
|| command || Command to run for the test. This is not required for manual tests, but it can be used, anyway. ||
|| depends || List of names of tests on which this test depends. So, if the other test fails, this test will be skipped. ||
|| requires || Registry expressions which are requirements for this test: 'input.mouse' in info.capabilities ||
|| timeout || Timeout for running the command. ||
|| optional || Boolean expression set to True if this test is optional or False if this test is required. ||
=== What plugins are available? ===
The list of currently available plugins is:
 * local: Include ''local'' test suites that have been written for checkbox.
 * remote: Integrate ''remote'' third party test suites that have been written for another framework (i.e. LTP, mago, ...)
 * manual: Write ''manual'' test cases
 * shell: Write a automated ''shell'' test cases
 * attachment: Add ''attachment'' to report, that is, important information that must kept for reference
 * resource: Gather hardware ''resource'' information
Line 82: Line 83:
==== Special fields ==== === What plugins do I really need to write tests? ===
For test suites: '''local'''. For test cases: '''manual''' and '''shell''' (for non-interactive tests).
Line 84: Line 86:
 * $output: This will be changed by the output of the command. Example:

{{{
command: date
description:
 was uds good for you on $output?
}}}

In this case, the description will show the current date. Clicking on "Test again" will re-render the description.

=== Test Suites ===

Tests can be grouped by suites. A suite is basically a text file that contains one or more tests.
Suites are available under the directory: /usr/share/checkbox/suites

All the files under that directory are automatically parsed for tests.

Test suites are parsed by the suites_info plugin (check the Plugin section for more information about plugins)

==== Environment variables ====

Apart from the two options explained above, this is the place to define the environment variables that we want for our tests. From the example above this would be:

data_path = %(checkbox_directory)s/data

=== Creating Tests and Suites ===

For a more detailed walkthrough of creating checkbox tests and suites, see [[Testing/Automation/Checkbox/Walkthrough|this walkthrough]].

=== Testing your suites in a bzr checkout of checkbox ===

If you are adding new suites to checkbox, you can simply test them from within the checkout, you can test directly from there:

{{{
./bin/checkbox-gtk --config="checkbox/plugins/suites_info/whitelist=mysuite.txt.in"
}}}

=== Easiest way to run your test with Checkbox ===

If you don't need any particular parsing of the results and you only want to run your test with Checkbox all you need to do is:

 * Prepare one or more scripts to perform the test.
 * Place the script(s) in /usr/share/checkbox/scripts.
 * Add the test to an existing suite (or create a new suite) in /usr/share/checkbox/suites.
 * Run checkbox.

=== Test results ===

Test results are sent automatically to Launchpad if an internet connection is available.
Apart from sending them, they are stored temporarily at /var/cache/checkbox. This is the default folder, but it can be changed with the environment variable ''CHECKBOX_DATA''.

To change the results folder in run time just start the application with:

{{{
CHECKBOX_DATA=/tmp sudo -E checkbox-gtk
}}}

== Plugins and Registries ==

Plugins and registries control how checkbox functions, executes scripts, and gathers information. While they are fully configurable and extensible, most tasks can be accomplished with the existing plugins and registries.

=== Plugins ===
A large number of plugins are provided by default, and can be seen in /usr/share/checkbox/plugins. When writing tests, the most common plugins used will be ''manual_prompt'' (for manual tests) and ''shell_prompt'' (for shell or non-interactive tests).

==== Description ====
Plugins change Checkbox default behaviour. In fact, the checkbox default behaviours are plugins themselves.

==== Default Plugins ====

==== Implementation ====
=== Implementation ===
Line 180: Line 113:
=== Registries === === Resources ===
Line 182: Line 115:
==== Description ====
Registries provide a consistent way of accessing hierarchical information in the system. They are meant to expose a single rooted data structure which can be accessed consistently regardless of where the information might be stored.

==== Contents ====
The existing registries give fast access to a large amount of system information. This includes:
 * CPU information
 * PCI and USB bus information
 * HAL information
 * Installed package status (dpkg)
 * GConf contents
 * Kernel modules
 * Drive mount information
 * Extended system information (through sysctl)

Registries can be queried via a simple text search within a suite.

==== Implementation ====
One of the motivations for providing this kind of data structure is to be able to define the requires: field in tests in a consistent way. Another is to provide plugins with a convenient way to obtain and query information about the system.

A registry is simply a Python module that extends the Registry class. A minimal registry would look like:

{{{
class Cr3Registry(Registry):
   
   def items(self):
       return (("foo": 1), ("bar": 2))

factory = Cr3Registry
}}}

The new Registry simply needs to define an items method and then the RegistryManager handles the rest.
TBD

Introduction

Checkbox (https://launchpad.net/checkbox) is a test runner for Ubuntu. It aims to provide a common framework to run all types of tests, from hardware tests, to command line tests, unit tests or desktop tests and send their results to Launchpad, automatically.

General Configuration

Checkbox general configuration is stored at /etc/checkbox.d. Each way of calling checkbox (GUI, cli) has its own configuration file.

checkbox.ini

This is the general config file. Properties listed here will be inherited by all checkbox flavours.

Example:

[DEFAULT]
version = 0.9.1
plugins = checkbox/plugins

[checkbox/plugins]
modules = %(checkbox_share)s/plugins/*.py

Explanation of options:

  • [DEFAULT] - This section covers all general configuration properties.

  • version - The version of checkbox to use the configuration with.

  • [checkbox/plugins] - This section controls the checkbox plugin system.

  • modules - The path to the checkbox plugin executable scripts.

Jobs

What is a job?

A job is a task or unit of work that is used to define many different things including test suites and test cases. All jobs are composed of a list of fields that have a special meaning.

For more information about writing jobs, please refer to the following sections:

What fields may a job include?

A job can include any of the following fields:

  • name: Unique identifier for a job
  • plugin: Takes care of the interpretation of the job
  • depends: The the job cannot be executed, at least until another one has been executed successfully
  • requires: Set of requirements that must be satisfied for the job to be executed. Typical requirements are of the kind some package must be installed or some hardware must be detected.

  • timeout: How much time should checkbox wait before cancelling job execution
  • command: A string that must be executed using the same commands as in a shell
  • user: Don't execute command with the same user that launched checkbox, but with a different user (usually root)

  • description: Human-readable description of the job purpose

Are there any mandatory fields?

Yes, name and plugin must always be present. Depending on the plugin, description and command may also be required. However, it's advised to use description in every job (even when not needed) for documentation purposes.

Where are jobs stored?

In source code you'll always find jobs under jobs directory. In an installed package, such as checkbox, the jobs are under /usr/share/checkbox/jobs directory.

Which format is used for jobs?

Jobs are stored using rfc822 format. Despite this might not seem a very convenient format, it's really useful because it supported by the translations tools and this makes possible to translate jobs easily.

What's the difference between .txt.in and .txt?

On one hand, in source code you'll find that jobs use the .txt.in extension. This is because in they are translations templates that contain extra information information about which strings are expected to be translated.

On the other hand, in the package you'll find that jobs use the .txt extension. This is because they are just job description files and the information for translations has been processed and stored in special files.

Why _description instead of description?

In rfc822 files, the underscore prefix is used to mark a field as translatable. Hence all .txt.in files use this prefix to mark description as a translatable string.

Plugins

What is a plugin?

A plugin, in the checkbox vocabulary, is a script that takes care if the interpretation of a job, that's to say, it implements the policy that it's going to be applied to that job.

What plugins are available?

The list of currently available plugins is:

  • local: Include local test suites that have been written for checkbox.

  • remote: Integrate remote third party test suites that have been written for another framework (i.e. LTP, mago, ...)

  • manual: Write manual test cases

  • shell: Write a automated shell test cases

  • attachment: Add attachment to report, that is, important information that must kept for reference

  • resource: Gather hardware resource information

What plugins do I really need to write tests?

For test suites: local. For test cases: manual and shell (for non-interactive tests).

Implementation

A plugin is simply a Python module which registers for events. A minimal plugin would look like:

class Cr3Plugin(Plugin):
    def register(self, manager):
       super(Cr3Plugin, self).register(manager)
       self._manager.reactor.call_on("report", self.report)
       
    def report(self):
       self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

factory = Cr3Plugin

A plugin should inherit from the Plugin class and it should register to be called on particular events. In this example it is registered to call the report method upon the "report" event.

self._manager.reactor.call_on("report", self.report)

The report method itself fires the "report-cr3" and anyone who has registered for this particular event will be called.

self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")

The plugin should provide a global factory variable which will be the hook for instantiating your plugin.

Resources

TBD

Tips And Tricks

[/TipsAndTricks]

Testing/Automation/Checkbox (last edited 2021-10-20 11:49:54 by sylvain-pineau)