Testing/fix-validation-tracking

fix-validation-tracking

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

Launchpad Entry: fixed-bug-verification
Packages affected:

Summary

Use a combination of structured entries in Launchpad and scripted web pages to track certain categories of high impact bugs including SRU candidates and fixes applied very close to release. These should have test cases and receive structured testing.

Rationale

These bugs are currently tracked in a range of places*, none of which do everything we want. A better overview will make it easier to track SRU testing and make it easier for new contributors to get involved.

* FixesToVerify, pending-sru

Use Cases

LTS point releases: Each bug fix scheduled for an LTS point release needs a btest case, regression testing and fix verification. The new tracking system will give an overview of the state of testing and easy access to details.
Late fixes: Firefox bug 144326 was fixed two weeks before release. With any code change this close to release we should test that the fix worked and doesn't cause other breakage.
Package maintainer and casual contributors can use these tracking pages to find targets to work on.

Design

Information about the state of a bug's verification will be stored in Launchpad on the bug itself in the description and in tags. This data will be parsed by a script producing a tracking page at verification.qa.ubuntu.com/<page>, where <page> is one of several tracking pages including main-sru, universe-sru, sru-archive.

The bug description for each SRU bug will contain:

A manual test case on the form:

TEST CASE: 

PROCEDURE:
  1.  Open a terminal.
  2.  Create an user by typing: adduser foo
  3.  Create another user by typing: adduser bar
  4.  Delete an user by typing: deluser bar
  5.  Try to log in with both users.

RESULT ON PASS:
  1. You are able to log in only with foo since bar account was deleted.

RESULT ON FAIL:
  1. You still can log in with both users (foo, bar).

END TEST CASE

Instructions on using the tracking system:

This bug is tracked as an SRU-candidate at: http://verification.qa.ubuntu.com . The bug description should contain a test case on a standard format and successful fix verifications should be marked with comments containing the string: fix-verified, and a failed verification by fix-verification-failed.

Implementation

The SRU tracking will be implemented as a python script producing a static HTML page every few minutes, powered by bughelper queries. A beta is available here.

Bughelper parses the description to find a test case, the tags for status and the comments for notification of passed or failed tests.

CategorySpec