Ubuntu Wiki

• Launchpad Entry: smoketesting

• Created: 2008-12-04

• Contributors: davmor2, heno

Summary

Start a routine of regular smoke-testing from debian import freeze to release to catch problems early.

Rationale

The Current Plan:
We do a batch of automated testing and some manual testing before each milestone.
The problem with this is it leaves little or no time for QA Team/Developers to track issues and resolve them.

1. It's important that all the Canonical supported apps function correctly if they don't it looks bad on us and the developers.
2. It's important that these issues are dealt with before release where possible.

Use Cases

End user starts up their machine installs Ubuntu and x app fails to start.
End user installs Ubuntu and it doesn't work with their gfx card.
End user goes to check email / web page but can't connect to the web.

Assumptions

You have a spare test machine or a vm on your regular machine you can keep up-to-date with the latest daily version of the development release of the desktop.

This goal can be achieved either by a fresh install each day or by simply upgrading using the update-manager/sudo apt-get update && sudo apt-get upgrade.

Design

The best time to start I believe would be after Debian Import Freeze.

Reasoning:
1. All major importing from debian should of finished at this point.
2. Only major upgrades should be imported after this date all of which we should know about from UDS/team meetings.
3. It give us and the developers the maximum amount of time to test, locate and repair issues.

Testing 1 app per day doesn't take that long on the whole and will cover all the apps in time for the next milestone release.

Implementation

To achieve this goal I believe the easiest way is for everyone in the QA team to test just 1 application a day (maybe 2 depending on just how many people do this). This is a full test of everything in the app so we know that all the functions work correctly. In most cases this will take a few minutes but on the whole the more complex apps will only take about an hour.

UI Changes

There should be some way to sign up to this process and also a way in which to track who is testing what so tests don't overlap.
Temporarily this could be done via the wiki but it would be good if this could be integrated into the test tracker further down the line.
Also a wiki page for developer to track the bugs we locate.

Code Changes

A tag in LP or a single wiki page that lists the bugs located during the testing process that the developers can look at for fixing issues.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

BoF agenda and discussion

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

CategorySpec