Summary

The Canonical QA Team has utilized checkbox in order to allow for manual tests to be run, and the results aggregated for community testing of ubuntu phones.

At the moment, we're evaluating two possible ways to run these tests. The first is utilizing the PC and phone, with the phone tethered to the PC. The second uses only the phone itself to test. We are interested in feedback on which approach is easier for you as a tester.

Please try both approaches and provide feedback on which is easier

I found a bug / This part should work differently / It would be great if this had feature X

Awesome! Please file a bug and let us know!

https://bugs.launchpad.net/ubuntu-community-testing

Getting the tests

Option1: Using the PC

For now, everything you need can be found in the community testing ppa. Install the ppa and the community testing package.

sudo add-apt-repository ppa:ubuntu-testcase/community-testing
sudo apt-get update
sudo apt-get install ubuntu-community-testing

Option 2: Using only the phone

For evaluation purposes, consider this would merely be a click you are installing from the store. For now, do it manually

Download and install this click

http://people.canonical.com/~brendan-donegan/com.ubuntu.checkbox_1.2.1_multi.click

adb push com.ubuntu.checkbox_1.2.1_multi.click /home/phablet
adb shell sudo -H -u phablet pkcon install-local --allow-untrusted com.ubuntu.checkbox_1.2.1_multi.click

Running the tests

Option 1: Using a PC

Typically an announcement will go out asking for results to be gathered against a specific testset. If you don't have a specific testset to test, or simply want to run results, you can leave off the argument to run against the most recent testset.

ubuntu-community-testing --testset=<testset number>

or simply;

ubuntu-community-testing

NOTE: In order for the test results to be accepted, you should plug your device into your PC as the sole device available over adb. You will need developer mode enabled on your device as well

Option 2: Using the phone

Find and launch the checkbox application you installed.

Checkbox

After starting, checkbox will load. Press the start testing button.

Option 1: Using the PC

Select uncategorized.

Option 2: Using the Phone

Select Practitest. Then select uncategorized.

Both

Select any/all of the tests in the testset, and run them.

NOTE: You need to run the system-information test which will gather information from your phone automatically. Your submission will be rejected without it. The test will run automatically without input from you, just be sure to leave it selected.

You can add comments during the run. Select Yes (if the test passed) or No (if the test failed) to be moved onto the next test.

Submitting Results

Option 1: Using the PC

Once the testing is complete a summary screen and graph will be displayed. Select to Upload Results. Checkbox will prompt for your Ubuntu SSO login details and submit your results.

Option 2: Using the Phone

The click built for testing you installed above is missing the upload code. For now to evaluate the option, assume it will work the same as on the PC.

Viewing the community's testing

For now, the aggregated results from everyone in the community won't be visible. Work is continuing to make the results viewable in realtime. Until then, know your testing is helpful!

Thanks for helping test!