StructuredTestingPrograms
Launchpad Entry: qa-m-community-structured-testing
Created: 2010-04-26
Contributors: apulido
Packages affected: None, but it may affect the ISO tracker
Summary
uring the Lucid development cycle we run a testing program for proprietary graphic drivers and their integration with Ubuntu. Details of the program can be found at the wiki https://wiki.ubuntu.com/X/Testing/ProprietaryDrivers/WeeklyProgram.
Feedback from participants https://wiki.ubuntu.com/X/Testing/ProprietaryDrivers/WeeklyProgram/Feedback was very positive and some of them are even willing to organize similar testing efforts.
We will gather feedback to improve this kind of initiatives.
Release Note
This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)
It is mandatory.
Rationale
This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified.
User stories
Assumptions
Design
You can have subsections that better describe specific parts of the issue.
Implementation
This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:
UI Changes
Should cover changes required to the UI, or specific UI that is required to implement this
Code Changes
Code changes should include an overview of what needs to change, and in some cases even the specific details.
Migration
Include:
- data migration, if any
- redirects from old URLs to new ones, if any
- how users will be pointed to the new way of doing things, if necessary.
Test/Demo Plan
It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.
This need not be added or completed until the specification is nearing beta.
Unresolved issues
This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.
BoF agenda and discussion
Agenda
- Explanation of the project to those who didn't know about it
- Infrastructure
- The ISO tracker: what worked, what didn't and what's missing
- The testcases wiki
- The design team use case format. How can we use it?
- Communication
- Mailing list
- Forums
- Documentation
- Community engagement
See also RedHat's Bodhi - http://fedoraproject.org/wiki/Bodhi_Guide - process which utilizes tester community to test packages before they're put into the distro officially (analogous to our *-proposed and SRU processes, but which includes structured testing and package testing karma).
Discussion
# Explanation of the project to those who didn't know about it Structured testing, executing test cases throughout the development cycle https://wiki.ubuntu.com/X/Testing/ProprietaryDrivers/WeeklyProgram Daily reports http://qa.ubuntu.com/reports/xorg_prop_drivers/ Coordinated through Launchpad team and mailing list # Infrastructure * The ISO tracker: what worked, what didn't and what's missing * The testcases wiki # The design team use case format. How can we use it? Output of design use case is acceptance criteria Have a single workflow starting with design team's use cases, flowing through to development and then QA They also have a Google Docs-based tool for editing and viewing use cases # Communication People found that having the opportunity to communicate with the devs was a very positive experience. Responsive developers-tester communication is a clear benefit of working with testcases rather than reporting bugs and waiting for feedback. Need to keep the simplicity of the testcases in order to make it easy for people to start testing and maintain focus * Mailing list * Forums * Documentation # Community engagement Problem: reporting - Overlap of tests being done on qa.ubuntu.com and checkbox tests - many listed here: http://testcases.qa.ubuntu.com/Hardware - avoiding duplication by having checkbox submit info to the QA tracker. - Check box has modular reporting - a plugin could be written to report to additional locations (current launchpad or certification website) - Assembly-line type testing which includes design teams use case mapper, and QA testcase tracker. - Use of project Sikuli can assist with automation of tests - http://groups.csail.mit.edu/uid/sikuli/ - Different information Either an individual or checkbox needs to verify the iso when testing. During lucid, sometimes the image did not change for 5-7 days before anyone knew it.
Specs/StructuredTestingPrograms (last edited 2010-05-10 14:04:33 by 217)