CrowdsourcedTestingSpec

This specification will outline how a system could be set up to allow any Ubuntu user to volunteer their system to run tests developed by Ubuntu QA and Developers. It could be run on stable releases and unstable releases, allowing tests to specify which versions to run on.

This spec will use Hudson, a free and open-source continuous integration server, as an example framework.

User Stories

• Bryce has an idea to improve graphics performance and wants to run his test on 10 different nVidia machines. He'd like to be able to rapidly iterate on his idea, so getting quick feedback is important.
• Ara believes an intermittent bug has been fixed in gedit and would like to run the same test 10 times on 20 machines for a total of 200 runs.

Requirements

• Trivial to join for users
• Straightforward for developers and QA to send tests to nodes and interpret the results

Joining the cluster

A user is running Ubuntu and decides they would like to contribute to Ubuntu by donating their computer to testing for a period of time. They visit crowd.ubuntu.com and see they can click a button to join their computer to the cluster:

When the button is clicked, a small application is run (via Java Web Start in the case of Hudson) which pops up a small window telling the user they have joined the cluster:

At any time they can close this window and no longer be part of the cluster.

Automated tagging

When a node joins the cluster, a job will run on it and generate some tags on the node such as "amd", "nvidia", "broadcomwl", "ssd", "karmic", "ubuntu-desktop", et cetera, allowing future jobs to specify a set of tags and get relevant nodes.

Running tests on the cluster

For a simple test, a developer can write a bash or python script directly in the Hudson interface:

For more complicated or existing scenarios, Hudson can check out code from svn/bzr/git and run it, so existing test suites could be easily across the cluster.

Whoever creates a specific job can configure how they will be notified of a success or failure, such as email, jabber, IRC or checking the web:

Integration with Ubuntu workflow

Developers/QA

Deploying a test to the cluster should be able to fit into the normal flow of writing a test to be integrated into Checkbox or writing a Mago test. One way to accomplish this would be to have a tag used in bzr commits. Checkbox tests seem to have a rich set of metadata, perhaps committing a new checkbox-compatible test with something like "#crowd" in the commit message would trigger that test to be pushed out to matching nodes, pulling the labels from the test information.

Checkbox UI

Perhaps opting in to cloud testing could be accomplished by launching Checkbox and clicking a button there which would make the node available to the cluster until Checkbox is closed.

Rewards

Users should be able to achieve some sense of reward by running tests in this and other fashions. Perhaps awarding the user one point for each test run on their computer, with a trend graph for each user (to encourage velocity/acceleration), and a leader board available on qa.ubuntu.com.

One thing this would have to handle is a user re-running the distribution checkbox tests over and over.