Structured Testing Programs (STP) are a formalized way to engage the community to test a particular feature, regression potential, updates, etc.
During the Lucid development cycle, the STP to test the proprietary X drivers was succesfully run during 10 weeks.
Based on that experience, this is a how to create a STP, step by step.
Things to set up
First of all you will need a wiki page to store the documentation of the STP. Normally you would want to host the wiki page under http://wiki.ubuntu.com/Testing/NameOfTheProgram, but you are free to host it some place else.
You will base your wiki page in this template.
The goal of your STP
In order for the STP to be successful you need to have a clear goal. Why are you creating the STP? What do you want to be tested? What you don't want to be tested? (or what is not a priority). If your scope is too wide, people might feel overwhelmed.
Testcases are devided in the description of the testcase itself and the results tracker.
In the testcases wiki we provide the description of the testcase. The instructions of each of the testcases should be easy to follow, and will have a deterministic pass/fail condition. The tester should know, at the end of the testcase, if the test passed or failed.
The syntax of the testcases are described in the http://testcases.qa.ubuntu.com/CaseAndPlanGuidelines. You can browse around the site to check some examples based on existing testcases.
The Results Tracker
For tracking results we use ISO tracker instances.
This is the current list of instances that we have set up:
You have to select one of these sites to host your testing. If there is no one that matches your testing, you will be able to open a new one (if accepted).
Once you have selected, you will need to send an email to Ara Pulido with the following information:
- URL of the wiki page
- Tracker you want your STP to be hosted
- A text file with the following format (each line is a testcase):
|| Product || Testcase Title || Testcase URL ||
where Product is the name of the product you want to test (i.e. Firefox, Ubuntu Desktop, gEdit, etc.), Testcase Title is a title for your testcase, and the URL is the location of the description in the testcases wiki.
Setting up Instructions
You have to provide your testers with clear instructions on how to set up their testing environment. Some questions you may need to answer:
- Which Ubuntu version should your testers be running?
- Which packages (PPA? official?) do they have to install?
- Is it necessary to test on bare metal, or a VM is OK for your STP?
You also have to provide with instructions on how to set up an account in the test tracker. Most of this information is already provided for you in the template, but make sure that you edit it with the correct tracker URL.
Apart from following each testcase, in this section you have to answer how to perform the testing. Is it a weekly testing? Is there any specific dates to start the testing? How do people use the tracker.
In the template you will get the basics of using the tracker, but you may want to customized it by changing the screenshots.
In this section you will explain how to report the issues people may find during their testing. You will link to any previous documentation on how to report issues and specific debugging instructions for your topic.
Also, if you are testing against a PPA rather than an official Ubuntu package, you may want testers to file bugs against a different project (i.e. the upstream project).
Launchpad team and mailing list
If the STP is going to take long (it may run throughout the developement cycle, or you may want to run it every cycle) it is a good idea to set up a specific Launchpad team and an associated mailing list. If they are part of a team and they receive emails in a specific mailing list, testers feel engaged with the project, and they are more likely to finish their testing.
It is also important to engage the developers that are relevant to the STP. Make sure the join the mailing list and they participate in the discussion. The best compensation testers can have are seeing that the bugs they are feeling and the problems they are having are taken care of by the developers.
Call for Testing
This is an important bit in your STP: the Call for Testing.
The Call for Testing takes the form of an email, blog post, tweet, etc (the more the merrier). There isn't a recipe that always work, but there are some tips that may help reaching the right audience.
- The subject always need to have a "Call For Testing/Help: Topic". If you only put the topic and not "Call For Testing" people may ignore the email if they are not interested on the topic.
- When writing the email use always only-text emails (work better in mailing lists) and put every URL in a separate line and tabbing it fours spaces.
- In the first paragraph explain breafly what kind of testers are you looking for and why are you calling for testing. Again, briefly.
- The rest of the email should be just a list of things to have or to do in order to help. Do not write long paragraphs, just a numbered list of steps.
You should be sending your email to ubuntu-quality, ubuntu-devel as well as any specific to your topic mailing list (i.e., if you're testing graphic drivers you may want to send it to ubuntu-x as well). This is an example of a Call For Testing email:
Call For Testing: ATI and nVIDIA graphic cards Do you have a *nVIDIA* or *ATI* graphics card? Do you want to help ensure users have a smooth experience if they choose to use the proprietary drivers? We are looking for committed volunteers to test nVIDIA and ATI proprietary drivers on a weekly basis. The goal of this testing is to catch regressions early in the cycle, and fix bugs before they reach a major audience. If you want to be part of the team you will need: 1. A computer with an ATI (R600 or newer) or nVIDIA (GeForce series 2 or newer) graphics card 2. A spare partition on that system * If you don't have a spare partition you can easily create one. 3. One hour of your time per week 4. An Internet connection If you want to take part in this adventure, please, check the detailed instructions in the wiki: https://wiki.ubuntu.com/X/Testing/ProprietaryDrivers/WeeklyProgram Thanks for helping making Ubuntu even better! P.S. This project is to test the proprietary drivers. If you're interested in testing the free drivers, we don't need installation testing but help is always welcome. Check how at the Ubuntu X team page .  https://launchpad.net/~ubuntu-x-swat