This document came from the Italian experience in setting up a LoCo Testing Team. If you're a LoCo Leader who made the same experience you're welcome to add your thoughts. If you have doubts in changing this document get in touch with Paolo Sammicheli.
The Italian Experience
The Italian Testing Team idea comes from a chat with Marjo Mercado during UDS Lucid in Dallas. The group started experiencing tests with Lucid Lynx during Alpha 2 release and finally become an official group in the LoCo Team for Beta1 release. We decided to perform ISO Testing with Live testcase because we wanted a very easy and low requirement testcase for starting the group. This experience was discussed during UDS Maverick in Bruxelles (link to the blueprint) with the goal of to improve it and to find how to encourage other LoCos in setting up their testing team. In the meeting minute you will find data and thoughts about the Italian team.
Why a Loco Testing Team
ISO Testing is the best first-activity for a newcomer. In fact, testing is what every LoCo Leader always desired: a fast, easy task to give at those who want to start contributing but they don't know what to do because they do not have enough experience in computers. It's a healthy way to start because while testing Ubuntu you start using and learning the tools of the community (Launchpad, Wiki, Iso Tracker, etc), you learn how to report effective bugs and you learn new things about Ubuntu: how is made, who made it, where all these programs came from, etc. In the Italian experience, people who started contributing with testing, later started to contribute in other ways: triaging, documentation, translations, etc. It's very important to catch people's enthusiasm in participating in Ubuntu by giving concrete goals to reach from the beginning. On the other side it's really a wide activity. While it could be seem as easy to perform some test-cases, you will need a lot of experience and a deep knowledge of the release's goal to perform really valuable tests and make useful and clever reports. Testing is not a task you can learn in just one cycle and it also requires keeping up to date with what is happening on the development side. On the other hand, this involvement in building the release keeps the activities really fun and interesting along the way.
Setting Up the Team
To start you need a core team, 2 or 3 of the most active people in your LoCo should work. In the Italian experience I can say that people already involved often are too busy for testing. You need to find active people but with a little of time still free. Maybe you will find people that know Ubuntu well but not the Testing Topic. Performing tests on our own for a few weeks while learning testing is a best practice that helped us a lot.
When you have a few people for the stars of your core team and you are confident they know Ubuntu testing quite well for tutoring newcomers, you should create the group. You will need some Wiki pages with descriptions of the group and the testcases you want to adopt. You will find the adoption topic in next chapter. During the Italian experience we opened a separated mailing list using Launchpad. We decided to keep being part of the team and subscribing to the mailing list separate. We encourage everybody to subscribe to the mailing list but we have a few minimum requirements for being a member of the team (see Team membership). An IRC channel can also help to keep the team in touch and ask for suggestions. In summary, at the moment of writing this document the Italian Team has the following tools:
- Launchpad Team
- Mailman Mailing List
- Wiki Pages
- IRC Channel
Team's board on the LoCo Forum
If your LoCo has a "how to contribute" page, you will want your team to be listed there too. We also had a sticky notice in our forum for a while which announced the call for participation to the team.
To keep people motivated and to make them feel part of the team we introduced the Adoption concept (see image below. Original page available here)
We intend Adoption to be the minimum commitment for a contributor. It makes the tests across the flavors very reliable for the QA Team and gives everybody a role in the team. We invite all members to perform more tests than those adopted keeping it like just a minimum commitment.
In order to gain Group membership we ask to have the following in order: a properly compiled launchpad profile, a personal wikipage and signing code of conduct. Then we ask new members to read the pages about the testing and to adopt at least one ISO image. We encourage newcomers to find the less well covered images but we leave them the freedom of choosing which one. In order to remain a member of the group we ask members to test each milestone. At the moment of the writing of this document we haven't yet decided on strict rules about expiration from group but we have an acknowledgement that in order to remain a member people must perform at least the adopted tests every milestone.
Break the Language Barrier
If you're a native English speaking LoCo this paragraph would not sound really interesting for you. I suggest you read it anyway because LoCos are spread all around the world and language is an issue.
If not a native English speaker, the first problem a newcomer faces in contributing to Ubuntu is the language. Translating the instruction for performing tests in their native language helps to better understand the process. Don't underestimate this aspect. Also for those who do know some English, reading a page in their own language is less tiring than in English. Prerequisites to starting to contribute is a tiring process: registering on Launchpad, Wiki, Code of Conduct, etc. Making the reading more comfortable helps a lot.
A few pages that we found useful to translate are:
Mentoring newcomers for testing has been shown to be very demanding. The first step to being involved in Ubuntu includes all the difficulties in learning the tools: wiki, mailing lists, irc, iso tracker, launchpad, etc. As a result, if you delegate the mentoring to the mailing list you will get a lot of traffic. After a while the Italian Team decided to organize a steering committee who operates through the IRC Channel for helping newcomers. We focus on the quality of the answers that the committee provides and we organized a FAQ page maintained by the committee itself. For access to the steering committee, people must be part of the team for at least one full cycle.
The Value of Testing
Although ISO testing is a good activity for newcomers, it is also release critical. If testing is not done correctly, it can result in a bad release. Also, any milestone can't be released unless all mandatory testcases are well covered and properly reported. While mentoring newcomers you should keep this in mind. Reports from newcomers in the beginning are less useful. Only once they know the process will their contributions be taken into account. Often to perform a good test you need to know the goals of the release. To be a good mentor you'll need to follow a few development and design mailing lists.
Marketing Side Effects
We notice that having more people involved in testing and aware about new features in the upcoming release creates a nice "marketing side effect". Many people start talking about nice things they discovered during the tests in their blogs, features they loved (and also they hated) which increases the buzz about the upcoming release.