Strategy

Introduction and Scope

This test strategy is meant to set up the scenery of where the QA team is at the moment and what the objectives are for the coming releases. We realize that getting to the level of excellence required by a project like Ubuntu is not easy and are willing to put together a strategy that will get us there in a reasonable amount of time and also that we can measure ourselves against. This is, by definition, a dynamic document that will need to be assessed and refined as work progresses. The stakeholders of this document are Product Management, Engineering Management and the community as a whole, all of which will play an instrumental role in the execution of this strategy. The owner of this strategy is the Distro QA team.

Aims/Objectives

  • Make Ubuntu enterprise ready to increase the confidence of the users during the coming releases
  • Improve the user experience
  • Improve non-functional characteristics (performance, memory footprint, energy consumption)
  • Reduce the amount of bugs that escape the development phases
  • Optimize the community efforts and offer the opportunity to learn industry standard testing to those individuals willing to help
  • Automate as many test cases as necessary and organizing the manual test cases in a manageable way
  • Avoid duplication of efforts and make every helping hand count towards the end goal
  • Establish metrics for better quality assessment
  • Run test cases on all possible configurations and new images

Current situation

The Good

  • UTAH allows us to set up new testing in a short time
  • The QA-dashboard and the reporting are good and people can assess the current status of ubuntu easily
  • Strong Ubuntu QA team eager to test images regularly
  • Problems with daily builds are fixed on a timely fashion

The Bad

  • Not enough automation
  • Metrics are not widely understood/used, maybe some new metrics should be established
  • Not well defined responsibilities
  • Not enough HW

How to get there

Precise objectives

Theme

Let's make quality or lack thereof visible to everyone.

See what we did for Precise.

Q-R objectives

Theme

Establish a standard way of automating for Ubuntu and start adding test cases towards full coverage.

See what we did for Quantal and Raring

S Release

Theme

Finally! Increase the number of automated tests that run/report results on regular basis

Ubuntu Automation Test Harness (UATH)

Objective

  • Keep up with the needs of new test cases

Actions

  • Continue to add new features to UTAH that can be used by everyone doing testing
  • Make UTAH capable of running autopilot test suites
  • Make UTAH capable of running upgrade testing
  • Make UTAH capable of running autotest test suites
  • Support provisioning/testing of Calxeda "highbank" nodes.
  • Expand phone/tablet capabilities

QA Dashboard

Objective

  • Make test results easy to look at and understand

Actions

  • Keep adding new views for new test cases

Adding new tests

Objective

  • Enhance Smoke testing
  • Enhance Kernel SRU hardware coverage
  • Enhance upgrade testing
  • Enhance non-functional testing

Action

  • Add more smoke testing
  • Increase the number of machines that are used for kernel SRU testing
  • Move upgrade testing to UTAH
  • Increase the test cases for memory and energy consumption testing

QA Community

Objective

  • Run the automated testing that the community created (autopilot test suite)

Action

  • Add a job to smoke testing that runs the community’s autopilot test suite daily

18 months objectives

There is a comprehensive set of test cases that we can run on daily builds and enables us to assess the quality of the builds

24 months objectives

We can add test cases as new features are being developed (we have caught up with our backlog)

QATeam/AutomatedTesting/Strategy (last edited 2013-04-16 15:43:03 by cpe-70-112-141-221)