Creating Checkbox Tests

Checkbox Overview

Before attempting to integrate a test into Checkbox, familiarize yourself with the Checkbox overview.

Analyzing a simple test

Let's begin with a simple test that is already present in Checkbox and see how it works. In order to make sure that a desktop system is working, we want to be sure that a mouse (or some other pointing device) is functioning. The easiest way to do this is simply to ask the user if their mouse is working. This is what Checkbox refers to as a manual test -- something that requires some user input to finish.

An example of a manual test requesting user input:

In the screenshot, notice that the user is being given instructions and asked if the test works correctly. Let's take a look at how this is implemented.

In /usr/share/checkbox/suites is a file called 'manual.txt' that contains the suite of manual tests to be run. Inside this file are descriptions of all of the different manual tests that can be executed by Checkbox. In this case, we're interested in the 'mouse' test:

name: mouse
plugin: manual
categories:
 laptop
 desktop
requires: 'input.mouse' in info.capabilities
_description:
 Moving the mouse should move the cursor on the screen.
 .
 Is your mouse working properly?

Each field listed here gives Checkbox information about the test and how to execute it. (A full description of the fields that Checkbox recognizes is available on the checkbox overview.) Let's go through the fields used here line-by-line:

name: mouse

This is just a name used to refer to this test internally. If a test is executed, this is how the pass/fail result will be listed. It will not be shown to the user.

plugin: manual

This tells Checkbox which plugin to use when executing the script. In this case, this is a test requiring user input, so we're using the "manual" plugin. For more information on plugins, see the overview description of plugins.

categories:
 laptop
 desktop

This field tells Checkbox what type(s) of system should execute this test. In this case, we're interested on testing the mouse on laptops and desktops, but nothing else. So if we execute Checkbox on a server system, for example, the mouse test will be skipped. For more information on categories (and other optional fields), see the overview description of optional fields.

requires: 'input.mouse' in info.capabilities

This field tells Checkbox what requirements need to be met before the test can be run. These requirements are usually established through the use of registries. Registries are information collected automatically by Checkbox about the system, and can be referred to in tests. For more information about registries, see the overview of registries. For now, it's enough to know that this line will cause Checkbox to look for the string input.mouse in its list of system capabilities, indicating that the system has a mouse installed.

_description:
 Moving the mouse should move the cursor on the screen.
 .
 Is your mouse working properly?

This is simply a free-form description of the test that will be shown to the user. Each line should be preceeded by a space so that Checkbox is able to tell when this field ends.

Creating a new test

Let's create a new test for Checkbox to execute, one slightly more complex than the mouse example. We will create a simple test to verify that CPU frequency scaling is supported and working on the system.

cpu_scaling_test

First we create an executable version of the test. As you can see attached, I have done this as a shell script, but any executable format can be used. Analyzing the test itself is outside of the scope of this document; for now all you need to know is that the script returns 0 if the test is successful and 1 if it fails.

Once we have the script, we need to integrate it into Checkbox. We can do this in a way that is very similar to the mouse test, by adding it to the manual test suite:

name: cpu_scaling
plugin: manual
command: cpu_scaling_test
requires: 'cpufreq' in cpu.capabilities
_description:
 Click the Test button to test the ability of your system's CPU to \
 adjust to different speeds depending on system load and power settings.
 .
 $output
 .
 Did the test report a successful result?

You can see that the structure of this test is similar to what we've already seen. But there are a few things worth noting:

The command field is executing a script, but there is no path specified. This is because Checkbox attempts to be very smart about looking for executables. It checks the standard scripts directory, /usr/share/checkbox/scripts, the user's path, and several other standard locations as documented in the checkbox manpage.

The requires field is different as well. In this case, we're checking to see if the system reports that CPU frequency scaling is supported at all. If the string cpufreq isn't listed as one of the CPU capabilities, this test will be skipped entirely. (Again, this is driven by the registry system. More information about registries is available on the overview description of registries.)

Finally, you'll see that we're including a variable called $output in the description. When the test is executed, this variable will contain the output from the test, and in this case will display it to the user.

This setup will work, but it's a little inelegant -- after all, we're performing the test but then asking the user to verify that it worked. Can't we just perform the test and get the result, not having to ask the user anything? Of course, the answer is yes!

Manual Test Example

plugin: manual name: myApp/test1 _description: a brief description

1. Do something

   • -> This happens!

2. Do something else

   • -> That happened!

General Formatting

• Make sure you have a blank line between each of your tests in your files
• Strip all trailing whitespace
• Ensure your file ends with a blank line

Automatic tests

To change this test from a manual test to an automatic one, we need make only a few simple changes:

name: cpu_scaling
plugin: shell
command: cpu_scaling_test
requires: 'cpufreq' in cpu.capabilities
_description:
 Test the CPU frequency scaling system.

You'll see that the plugin field has changed from manual to shell. This indicates that we are going to execute the command field in a shell rather than ask the user for input. Accordingly, we've also dropped the question from the _description field -- after all, the user won't see this field as they won't have to do anything!

It's good practice to separate tests into different suites based on functionality. In this case, it makes sense to remove this from the manual suite and create a new one, perhaps called cpu. If we develop new CPU tests in the future we can add them to this suite.

That's all there is to creating Checkbox tests and suites!