EdgyTestingSpec

Differences between revisions 3 and 4
Revision 3 as of 2006-07-06 21:12:18
Size: 3039
Editor: ottawa-hs-209-217-79-57
Comment:
Revision 4 as of 2006-07-07 20:12:55
Size: 5858
Editor: ottawa-hs-209-217-79-57
Comment:
Deletions are marked like this. Additions are marked like this.
Line 28: Line 28:
We will harness the efforts of the LaptopTestingTeam as well as the BugSquad to provide continuous testing throughout Edgy Eft's development cycle. In this plan, these awesome volunteers will be referred to as the "testing team".
Line 29: Line 30:
 * We begin by examining the [https://launchpad.net/distros/ubuntu/+specs feature specifications] that are scheduled for Edgy. These each need test plans, even preliminary ones, that will ensure basic sanity. We will have created a test plan for each approved specification by 20 July. This is meant to:
  * Ensure that each specfication has some form of measurable objective.

 * As each specification is worked upon, the testing team will periodically run tests on the partial implementations. Test plan development will continue in this stage, adapting as more test cases are thought up. New test cases should also be spawned whenever a new bug is discovered. Preliminary test results should be available by 17 August. This is meant to:
  * Measure the progress of each implementation in an objective way.
  * Catch regressions early in the development cycle.
  * Ensures that hugs are distributed early and often.

 * During the DistroSprint, the release managers will examine the test results and decide on which specifications to continue developing. And which to defer. This is meant to:
  * Refine which specifications we should concentrate our effort upon.
  * Concentrate developer effort on specifications that will complete successfully.

 * After the DistroSprint, the testing team will continue to refine and run the test plans for the remaining specifications. These test should provide firm data about these specs by FeatureFreeze on 7 September. The testing team will still run test plans all the way until FinalRelease. We expect that only bug fixes should be released into the archive, so test results should look more and more promising every day. This is meant to:
  * Track the progress of bug fixes.

 * After FeatureFreeze, a test plan for the installation will be written. Installer testing will begin in earnest on 28 September, which coincides with BetaRelease. We've done this kind of testing before for Dapper Drake, so this should be familiar to many people. Both the testing team and the DistroTeam will perform installer testing. This is meant to:
  * Ensure that the installer works on as many platforms as possible.
  * Test the installer on configurations that are unavailable to the DistroTeam.

 * Specification and installer test results for all architectures should be up-to-date for the ReleaseCandidate, on 19 October. This is meant to:
  * Give the release managers enough information to decide whether to perform the FinalRelease.
Line 53: Line 75:
|| 15 || September 28th || ||<#FFCCCC> /!\ '''BetaRelease''' /!\ '''UniverseFreeze''' || <!> Last new spec tests || || 15 || September 28th || ||<#FFCCCC> /!\ '''BetaRelease''' /!\ '''UniverseFreeze''' || <!> Test installer ||
Line 55: Line 77:
|| 16 || October 5th || ||<#E47A7A> /!\ '''KernelFreeze''', /!\ '''StringFreeze''' || <!> Test installer || || 16 || October 5th || ||<#E47A7A> /!\ '''KernelFreeze''', /!\ '''StringFreeze''' || ||
Line 57: Line 79:
|| 18 || October 19th || ||<#E47A7A> /!\ '''ReleaseCandidate''', /!\ '''LanguagePackTranslationDeadline''' || ||
|| 19 || October 26th || ||<#E47A7A> /!\ '''FinalRelease''' || <!> Final installer results ||
|| 18 || October 19th || ||<#E47A7A> /!\ '''ReleaseCandidate''', /!\ '''LanguagePackTranslationDeadline''' || <!> Final installer results ||
|| 19 || October 26th || ||<#E47A7A> /!\ '''FinalRelease''' ||  ||
Line 61: Line 83:

 * We need to write up templates for the Wiki:
  * TestPlanTemplate
  * TestResultsTemplate

Summary

This specification outlines our plan for getting Edgy tested properly.

Rationale

Previous releases have been tested in an ad-hoc fashion. Apparently, slightly more organization would be useful, although this has to take into account the organic and volunteer nature of most of our testing infrastructure.

User stories

  • Alexandria is a release manager for Edgy and wants to know if a feature should be kicked out of the release. She consults the testing results offered by the QA efforts and finds the right information to make a decision.
  • Barry is an Ubuntu user who wants to help test Ubuntu, but doesn't know where to start.
  • Carolyn is a Ubuntu user who is on the LaptopTestingTeam. She wants to help test Ubuntu and to report her results so that they'll be acted upon.

Scope

This specification outlines how testing is going to approached for Edgy. This is not meant to be a general specification stretching until the end of time.

Plan

We will harness the efforts of the LaptopTestingTeam as well as the BugSquad to provide continuous testing throughout Edgy Eft's development cycle. In this plan, these awesome volunteers will be referred to as the "testing team".

  • We begin by examining the [https://launchpad.net/distros/ubuntu/+specs feature specifications] that are scheduled for Edgy. These each need test plans, even preliminary ones, that will ensure basic sanity. We will have created a test plan for each approved specification by 20 July. This is meant to:

    • Ensure that each specfication has some form of measurable objective.
  • As each specification is worked upon, the testing team will periodically run tests on the partial implementations. Test plan development will continue in this stage, adapting as more test cases are thought up. New test cases should also be spawned whenever a new bug is discovered. Preliminary test results should be available by 17 August. This is meant to:
    • Measure the progress of each implementation in an objective way.
    • Catch regressions early in the development cycle.
    • Ensures that hugs are distributed early and often.
  • During the DistroSprint, the release managers will examine the test results and decide on which specifications to continue developing. And which to defer. This is meant to:

    • Refine which specifications we should concentrate our effort upon.
    • Concentrate developer effort on specifications that will complete successfully.
  • After the DistroSprint, the testing team will continue to refine and run the test plans for the remaining specifications. These test should provide firm data about these specs by FeatureFreeze on 7 September. The testing team will still run test plans all the way until FinalRelease. We expect that only bug fixes should be released into the archive, so test results should look more and more promising every day. This is meant to:

    • Track the progress of bug fixes.
  • After FeatureFreeze, a test plan for the installation will be written. Installer testing will begin in earnest on 28 September, which coincides with BetaRelease. We've done this kind of testing before for Dapper Drake, so this should be familiar to many people. Both the testing team and the DistroTeam will perform installer testing. This is meant to:

    • Ensure that the installer works on as many platforms as possible.
    • Test the installer on configurations that are unavailable to the DistroTeam.

  • Specification and installer test results for all architectures should be up-to-date for the ReleaseCandidate, on 19 October. This is meant to:

    • Give the release managers enough information to decide whether to perform the FinalRelease.

Timeline

See EdgyReleaseSchedule, which this is mirroring a bit.

Week

Date

Task

Status

Notes

June 2006

1

June 22nd

2

June 29th

July 2006

3

July 6th

4

July 13th

Warning /!\ UpstreamVersionFreeze, Knot CD 1

5

July 20th

Info <!> Test plans for new specs

6

July 27th

August 2006

7

August 3rd

8

August 10th

9

August 17th

Info <!> Preliminary test results

10

August 24th

Info <!> DistroSprint this week (proposed)

11

August 31st

September 2006

12

September 7th

Warning /!\ FeatureFreeze

Info <!> Firm test results

13

September 14th

Warning /!\ DocumentationStringFreeze

14

September 21st

Warning /!\ BetaFreeze

15

September 28th

Warning /!\ BetaRelease Warning /!\ UniverseFreeze

Info <!> Test installer

October 2006

16

October 5th

Warning /!\ KernelFreeze, Warning /!\ StringFreeze

17

October 12th

Warning /!\ NonLanguagePackTranslationDeadline

18

October 19th

Warning /!\ ReleaseCandidate, Warning /!\ LanguagePackTranslationDeadline

Info <!> Final installer results

19

October 26th

Warning /!\ FinalRelease

Unresolved issues

BoF agenda and discussion


CategorySpec

EdgyTestingSpec (last edited 2008-08-06 16:19:29 by localhost)