Launchpad Entry: lucid-qa-testing-team
Created: 2009-11-09
Contributors: AraPulido
Packages affected:
Summary
We need to increase community involvement in testing in earlier stages to minimize regressions due to specific hardware or configurations. We also need to improve the view of testing as a skilled process and a good way to learn more about Ubuntu and its development process.
Release Note
We have produced a roadmap for the team that is available at https://wiki.ubuntu.com/Roadmaps/Lucid/TestingTeam
Rationale
We cannot leave quality to good luck. We cannot rely in having millions of users who will find bugs as they use the applications. Our users want to use the software, not to find bugs and report them. FOSS projects in general and Ubuntu in particular need a new way of rethinking testing as a skilled activity and an opportunity to contribute to the project.
We want to build a Testing Team in Ubuntu to try to minimize the impact of bugs in the released versions. This team would have a mailing list and regular meetings on IRC. Activities will be diverse and will include things like: formal manual testing, exploratory testing, writing new test cases, organize and conduct community testing days, automated testing and developing new tools (yes, if you like to code, there’s also a place for you).
User stories
- Maria wants to start contributing with Ubuntu. She has read a bit about testing activities, but she really does not know where to start or who to communicate with. She goes to the Testing wiki page and she finds useful information about how to start contributing testing Ubuntu.
- Pablo is a usual contributor to the Bugsquad team. He triages bugs and participates in the bug days. He also likes other areas of QA, as testing, and he joins the testing mailing list. He finds a useful community that help him starting broadening his QA involvement.
Assumptions
- People are interested in helping testing Ubuntu, but they don't have a guidance on where to start.
Design
The design will be the different activities that the team will perform
Testing days
- Give it a specific focus (e.g. "test the proposed mesa update")
- Prepare "canned tests", and be sure to test the test, e.g.:
- report new bugs
- different day from bug day where you close bugs
- put two bug tags: testing-day and eg testing-day-dec25 so we can easily track all bugs that were a result of a test day (intrinsicly rewarding when they get closed)
- publish on Twitter/identi.ca
Wiki Pages
- Updated with testing news:
- "Testing project of the week"
- Recent major package updates
- ISO testing at appropriate times
- Areas with large numbers of bugs getting filed
- should mention: testdrive, iso testing, launchpad teams
Recognition of testers
- Ubuntu membership and Hall of Fame
- Launchpad karma? (we currently have no concept of test karma)
- Using a bzr branch to collect the data
BoF agenda and discussion
Spec at https://wiki.ubuntu.com/Specs/TestingTeam
Current situation (Problem):
- Have launchpad testing team with many many members but it's not a real team (no mailing list, meetings, structure, roadmap, etc)
Agenda:
* Objective of the team * Same team or a new one? * Mailing list * IRC meetings * Entry criteria
- Open team / Restricted team
- One team / v.s. Several teams
- eg ISO testing vs SRU testing could be separate teams
- single team provides advantage of being an entry point, however multiple teams provide more specific focus
- we desire a proliferation of community testing (eg kernel team wants one) in this cycle
- kernel team is using only bootable media for test kernels so users don't have to install anything
- Create a initial task for people who want to join/stay
- mention how to join in on the testing in the release announcements (so at eg Alpha 1 we can ask users to join the test team and run checkbox)
* Possible tasks (only in a very high level, as the objective is the creation of the meeting itself)
- Improve test cases wiki
- Maintain ubuntu-qa-tools
- Apport hooks
- Exploratory Testing (Session Tester)
TestPilot kind of application
- ISO Testing
* Recognition of testers
- Ubuntu membership and Hall of Fame
- Launchpad karma? (we currently have no concept of test karma)
- Let users see evidence that their testing makes Ubuntu better
- - Show their work has demonstratable value/impact - publish the reports somewhere (Launchpad?) - make them feel that the numbers are getting better - Give bugs filed by testers a higher priority
- Using a bzr branch to collect the data
- Bryce on mesa: didn't update mesa until they got enough test results (a game of sorts)
- Locos are a good candidate for doing testing
- Suggestion: rank locos or arbitrary teams according to testing
* TODO:
- - create an Ubuntu Testing product in launchpad - put a separate bzr branch for each kind of test - have individuals push to bzr branch each test result
- - they need to have their ssh key either on the testing machine or make the test data portable to a machine with the key
* how someone starts testing
- - recruited from top link in forums - in person from loco - community buzz - social technologies that we have (twitter etc)
* get sent to testing wiki page
- - this page currently needs to be much more actively maintained than otherwise
- - "Testing project of the week" - Recent major package updates - ISO testing at appropriate times - Areas with large numbers of bugs getting filed
* Get individual involved quickly. Easy to get started, quick to see the impact you're making, safe on your system. * Tutorial on "how to file good bugs from your test results" * Testing day
- - Give it a specific focus (e.g. "test the proposed mesa update") - Prepare "canned tests", and be sure to test the test, e.g.: - report new bugs - different day from bug day where you close bugs - put two bug tags: testing-day and eg testing-day-dec25 so we can easily track all bugs that were a result of a test day (intrinsicly rewarding when they get closed) - publish on Twitter/identi.ca
* The IRC meeting would be at the regular QA meeting (Wed 17UTC #ubuntu-meeting)
TODO:
- - turn on launchpad testing team mailing list - stgraber (done, waiting for approval) - update wiki page - Scott Ritchie - have someone maintain it consistently - Announce it (Fridge?) - ara - update test tools to put results into bzr branch - Ubuntu QA team blog: blog.qa.ubuntu.com