Launchpad Entry: https://blueprints.launchpad.net/checkbox/+spec/qa-maverick-hwcert-reevaluate-checkbox-flavors
Created: 2010-05-27
Contributors: fader
Packages affected: checkbox, checkbox-certification, checkbox-compatibility, checkbox-qa, checkbox-oem
Summary
Checkbox has fragmented over the past several cycles as different audiences find use cases for it. We should find a way to reduce or eliminate this fragmentation so that we have the minimum number of Checkbox packages required.
Rationale
Different audiences have found different uses for Checkbox. This is a Good Thing, as it means that the tool is a useful one. However, in addressing their needs, each audience has forked the tool and created their own "flavor". This is a Bad Thing as it leads to code duplication, confusion, and an unmaintainably large set of packages.
Ideally we should try to meet all the use cases for Checkbox within the main branch of the tool. This will allow us to reduce confusion, add new features in a more consistent fashion, and generally make the world a better place for system testing.
Use Cases
- Ender, an end user, wants to be able to test his system (or a system he might purchase) to verify that all the major pieces of hardware are properly working under Ubuntu. If a piece of hardware is not working, Ender wants to be able to file a bug against the appropriate package quickly and easily.
- Michael, a system builder, wants to be able to test the systems he is designing to be sure that all the components work correctly with Ubuntu so that it can be pre-installed on the system. If a piece of hardware is not working, Michael wants to be able to gather the appropriate information to send to the Ubuntu community and/or Canonical in order to get drivers written.
- Ronald, a member of the QA team, wants to be able to test a large number of systems simultaneously. He needs to be able to modify the tests to be run in a central location rather than visiting each system individually.
- Ara, another member of the QA team, wants to be able to test a specific component of a system in great depth. She also needs to be able to modify tests quickly and easily, and ensure that the same tests are being run each time.
- Sally, a retailer, wants to be able to test all the systems she sells with Ubuntu. She needs to design different test plans for each class of system in order to feel she has conducted appropriate tests.
Design
Modify Checkbox to enable it to read test definitions via a URL. This URL can be a file:// URL (for local storage), an http:// URL (for centralization), or any other schema supported by the system.
- Checkbox should fall back to the default tests if a URL is not provided.
- A versioning system must be supported so that future expansion is possible. (E.g. if we decide to enable Checkbox to pull more than simple test definitions via a URL, the system should be forward-compatible enough to allow this while also degrading gracefully in cases where it is not supported.)
This design addresses the above use cases:
- Ender is able to run the default set of Checkbox tests and get immediate coverage.
- Michael can run a webserver and add additional tests to test his specific hardware and gather information specific to his systems.
- Ronald can run a webserver and update his tests for all systems in one central location.
- Ara can keep a static file containing her tests and receive results for these tests on each test run.
- Sally can run a webserver that hosts several different test definition files. She can provide the appropriate URL for each system to have it retrieve the correct test definitions when tests are run.
Implementation
A new configuration setting will be added to Checkbox containing the URL that points to a test definition file. This setting should be supported both in a configuration file (e.g. /etc/checkbox.d/checkbox.ini) as well as through a command-line option passed to Checkbox.
To satisfy this specification, only a single file containing test definitions need be accepted. Future functionality may be implemented by additional specifications.
UI Changes
No changes to the main UI will be necessary. A configuration file change and a new command-line option will be available.
Test/Demo Plan
- Checkbox runs normally when no URL is provided.
- Checkbox retrieves test definitions from the URL provided and runs the tests from the retrieved file.
- Checkbox fails gracefully (with an appropriate error message) if the URL is unavailable or the retrieved file is unreadable.