RegressionTracking

Differences between revisions 7 and 23 (spanning 16 versions)
Revision 7 as of 2008-09-04 19:23:13
Size: 2407
Editor: c-24-21-206-172
Comment: add input
Revision 23 as of 2010-10-21 15:33:09
Size: 3896
Editor: ACaen-151-1-70-253
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
'''DRAFT:''' Procedure will be announced to the QA and dev teams when wrinkles have been worked out. Starting with Intrepid we are tracking functional regressions between releases. Regressions make for a poor user experience and therefore often count as release blockers. Even when we decide it is unavoidable to ship with a known regression we should document it clearly.
Line 3: Line 3:
Regressions will be marked on the Launchpad bug with one of these tags:
Line 4: Line 5:
Starting with Intrepid we will start tracking regressions between releases. Regressions make for a poor user experience and therefore often count as release blockers. Even when we decide it is unavoidable to ship with a known regression we should document it clearly. <<Include(QATeam/RegressionTracking/Table)>>
Line 6: Line 7:
Regressions will be marked on the Launchpad bug with one of 3 tags:  * Nominate / target to every release the regression found in whatever the release is(development or released).
 * Add a tag with the release name of the release the regression was discovered in first.
 * Add the last known good version to the description
Line 8: Line 11:
 * '''regression-potential''' - A bug discovered in the development release that was not present in the stable release. If it is significant, it should also have a task added under the [[https://launchpad.net/ubuntu-release-notes/ | ubuntu-release-notes]] launchpad project to be tracked for inclusion in the release notes.
 * '''regression-release''' - A regression in a new stable release. This may be a bug in a single package or functionality lost when changing the default application.
 * '''regression-update''' - A regression introduced by an updated package in the stable release.
Please also make sure that a comment is added to the bug when marking it with one of the regression tags, to match BugSquad policies.
Line 12: Line 13:
Regression that have been fixed should be marked with '''regression-fixed''' ''but the original regression tag should be left in place'' so we can track the evolution of these bugs. [or should we rely on bug history/regular poling /w archive?]
 * [Leann] - I don't think it will be necessary to tag bugs '''regression-fixed'''. The reason is that regressions should have already been tagged regression-*. Once a bug with a regression-* tag has a status change to "Fix Released" that should indicate the regression has been fixed.
 * [Brian] - I agree with Leann here
 * [Steve] - That makes sense. Should only bugs in main be nominated to the ubuntu-release-notes project, as well as requiring some level of severity (high?)?
 * [Leann] - Yes, I agree there should be a criteria in order to make the release notes - otherwise that section would likely become quite bloated and not as helpful. What Steve has described above seems fine. Should we also maybe take into consideration the number of subscribers or duplicates the bug has?
  
Line 29: Line 24:
== Example bugs ==
=== regression-release ===

||[[https://bugs.launchpad.net/bugs/200462|200462]] || Copying Files From CD/DVD Sets Permissions To Read Only || Hardy/Intrepid ||
||[[https://bugs.launchpad.net/bugs/290506|290506]] || cheese malfunctioning with UVCVIDEO webcams || Hardy/Intrepid ||
||[[https://bugs.edge.launchpad.net/ubuntu/+source/pgadmin3/+bug/610975|610975]] || relocation error with latest wxwidgets2.8 || Lucid ||
||[[https://bugs.launchpad.net/bugs/642518|642518]] || [MASTER] package fglrx 2:8.723.1-0ubuntu4 failed to install/upgrade: Kernel fix for CVE-2010-3081 breaks fglrx || Hardy/Jaunty/Karmic/Lucid ||


== Tracking ==

In addition to the tag search views in Launchpad itself, the regression tags will be monitored at reports.qa.ubuntu.com in a view similar to [[http://people.ubuntu.com/~sbeattie/regression_tracker.html|this prototype regression tracking page]]. Please report feedback on the comments section of this page or email [[mailto:sbeattie@ubuntu.com|sbeattie@ubuntu.com]] directly. There will also be an archive view.

== Special case tags ==

Some teams may want to track a certain type of regressions that do not fit into the scheme described here. An example is the recent move from the 2.6.26 to 2.6.27 kernel fairly late in the Intrepid cycle. In this case the kernel team was esp. interested in regressions from .26 to .27, tagging them [[https://bugs.edge.launchpad.net/ubuntu/+bugs?field.tag=regression-2.6.27|regression-2.6.27]]. These may not fit neatly into the tracking scheme described here (in this case a 2.6.26 was not shipped in a stable release so a regression from .26 to .27 is not actually a distro regression unless it also worked in 2.6.24), but we can still include such special cases on the tracking page.
Line 33: Line 44:


= Comments =

 * [Leann] - I don't think it will be necessary to tag bugs '''regression-fixed'''. The reason is that regressions should have already been tagged regression-*. Once a bug with a regression-* tag has a status change to "Fix Released" that should indicate the regression has been fixed.
 * [Brian] - I agree with Leann here
 * [Steve] - That makes sense. Should only bugs in main be nominated to the ubuntu-release-notes project, as well as requiring some level of severity (high?)?
 * [Leann] - Yes, I agree there should be a criteria in order to make the release notes - otherwise that section would likely become quite bloated and not as helpful. What Steve has described above seems fine. Should we also maybe take into consideration the number of subscribers or duplicates the bug has?
 * [Henrik] - I've removed the ''regression-fixed'' item

 * [jibel] [[https://lists.ubuntu.com/archives/ubuntu-devel/2010-September/031499.html|ubuntu-devel thread]]

Starting with Intrepid we are tracking functional regressions between releases. Regressions make for a poor user experience and therefore often count as release blockers. Even when we decide it is unavoidable to ship with a known regression we should document it clearly.

Regressions will be marked on the Launchpad bug with one of these tags:

regression-release

A regression in a new stable release or a development release. This may be a bug in a single package or functionality lost when changing the default application.

regression-update

A regression introduced by an updated package in the stable release.

regression-proposed

A regression introduced by a package in -proposed

regression-retracer

Used by the retracer when it finds a bug that has a similar trace to a previously closed bug.

  • Nominate / target to every release the regression found in whatever the release is(development or released).
  • Add a tag with the release name of the release the regression was discovered in first.
  • Add the last known good version to the description

Please also make sure that a comment is added to the bug when marking it with one of the regression tags, to match BugSquad policies.

Regression data template

It is important to use this exact template so we can programatically extract the information.

== Regression details ==
Discovered in version:
Last known good version:

Example bugs

regression-release

200462

Copying Files From CD/DVD Sets Permissions To Read Only

Hardy/Intrepid

290506

cheese malfunctioning with UVCVIDEO webcams

Hardy/Intrepid

610975

relocation error with latest wxwidgets2.8

Lucid

642518

[MASTER] package fglrx 2:8.723.1-0ubuntu4 failed to install/upgrade: Kernel fix for CVE-2010-3081 breaks fglrx

Hardy/Jaunty/Karmic/Lucid

Tracking

In addition to the tag search views in Launchpad itself, the regression tags will be monitored at reports.qa.ubuntu.com in a view similar to this prototype regression tracking page. Please report feedback on the comments section of this page or email sbeattie@ubuntu.com directly. There will also be an archive view.

Special case tags

Some teams may want to track a certain type of regressions that do not fit into the scheme described here. An example is the recent move from the 2.6.26 to 2.6.27 kernel fairly late in the Intrepid cycle. In this case the kernel team was esp. interested in regressions from .26 to .27, tagging them regression-2.6.27. These may not fit neatly into the tracking scheme described here (in this case a 2.6.26 was not shipped in a stable release so a regression from .26 to .27 is not actually a distro regression unless it also worked in 2.6.24), but we can still include such special cases on the tracking page.

Future development

Regression tracking should eventually become a Launchpad feature.

Comments

  • [Leann] - I don't think it will be necessary to tag bugs regression-fixed. The reason is that regressions should have already been tagged regression-*. Once a bug with a regression-* tag has a status change to "Fix Released" that should indicate the regression has been fixed.

  • [Brian] - I agree with Leann here
  • [Steve] - That makes sense. Should only bugs in main be nominated to the ubuntu-release-notes project, as well as requiring some level of severity (high?)?
  • [Leann] - Yes, I agree there should be a criteria in order to make the release notes - otherwise that section would likely become quite bloated and not as helpful. What Steve has described above seems fine. Should we also maybe take into consideration the number of subscribers or duplicates the bug has?
  • [Henrik] - I've removed the regression-fixed item

  • [jibel] ubuntu-devel thread

QATeam/RegressionTracking (last edited 2013-02-22 23:49:04 by javier-lopez)