Strategy
Introduction and Scope
This test strategy is meant to set up the scenery of where the QA team is at the moment and what the objectives are for the coming releases. We realize that getting to the level of excellence required by a project like Ubuntu is not easy and are willing to put together a strategy that will get us there in a reasonable amount of time and also that we can measure ourselves against. This is, by definition, a dynamic document that will need to be assessed and refined as work progresses. The stakeholders of this document are Product Management, Engineering Management and the community as a whole, all of which will play an instrumental role in the execution of this strategy. The owner of this strategy is the Distro QA team.
Aims/Objectives
- Make Ubuntu enterprise ready to increase the confidence of the users during the coming releases
- Improve the user experience
- Improve non-functional characteristics (performance, memory footprint, energy consumption)
- Reduce the amount of bugs that escape the development phases
- Optimize the community efforts and offer the opportunity to learn industry standard testing to those individuals willing to help
- Automate as many test cases as necessary and organizing the manual test cases in a manageable way
- Avoid duplication of efforts and make every helping hand count towards the end goal
- Establish metrics for better quality assessment
- Run test cases on all possible configurations and new images
Current situation
The Good
- UTAH allows us to set up new testing in a short time
- The QA-dashboard and the reporting are good and people can assess the current status of ubuntu easily
- Strong Ubuntu QA team eager to test images regularly
- Problems with daily builds are fixed on a timely fashion
The Bad
- Not enough automation
- Metrics are not widely understood/used, maybe some new metrics should be established
- Not well defined responsibilities
- Not enough HW
How to get there
Precise objectives
Theme
Let's make quality or lack thereof visible to everyone.
See what we did for Precise.
Q-R objectives
Theme
Establish a standard way of automating for Ubuntu and start adding test cases towards full coverage.
See what we did for Quantal and Raring
S Release
Theme
Finally! Increase the number of automated tests that run/report results on regular basis
Ubuntu Automation Test Harness (UATH)
Objective
- Keep up with the needs of new test cases
Actions
- Continue to add new features to UTAH that can be used by everyone doing testing
- Make UTAH capable of running autopilot test suites
- Make UTAH capable of running upgrade testing
- Make UTAH capable of running autotest test suites
- Support provisioning/testing of Calxeda "highbank" nodes.
- Expand phone/tablet capabilities
QA Dashboard
Objective
- Make test results easy to look at and understand
Actions
- Keep adding new views for new test cases
Adding new tests
Objective
- Enhance Smoke testing
- Enhance Kernel SRU hardware coverage
- Enhance upgrade testing
- Enhance non-functional testing
Action
- Add more smoke testing
- Increase the number of machines that are used for kernel SRU testing
- Move upgrade testing to UTAH
- Increase the test cases for memory and energy consumption testing
QA Community
Objective
- Run the automated testing that the community created (autopilot test suite)
Action
- Add a job to smoke testing that runs the community’s autopilot test suite daily
18 months objectives
There is a comprehensive set of test cases that we can run on daily builds and enables us to assess the quality of the builds
24 months objectives
We can add test cases as new features are being developed (we have caught up with our backlog)
QATeam/AutomatedTesting/Strategy (last edited 2013-04-16 15:43:03 by cpe-70-112-141-221)