Ubuntu Wiki

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

• Launchpad Entry: fixed-bug-verification

• Packages affected:

Summary

High profile bugs that have been fixed during the development cycle or previous releases should be covered during ISO testing. Automate tests for some high-impact bugs and otherwise include coverage in the manual test cases.

Release Note

Rationale

During Gutsy cycle we started a document (FixesToVerify) with a list of bugs that were fixed during the development cycle for testing them during the ISO testing period. The procedure was manual and ad-hoc. We didn't collect positive feedback regarding whether the bug was fixed for the users/testers.

Use Cases

• Bug 144326 crashes firefox when trying to print. The issue was fixed two weeks before release and we should confirm that it is and remains fixed until release.

Assumptions

Design

A manual test case will be written for each targeted bug and added to the bug description text. Corresponding automated test cases will be written in appropriate cases (as the testing infrastructure becomes available)

Test case descriptions

Test cases should be added to the description of the bug report in a standardised format, e.g:

TEST CASE:

PROCEDURE:

1. Open a terminal.
2. Create an user by typing: adduser foo
3. Create another user by typing: adduser bar
4. Delete an user by typing: deluser bar
5. Try to log in with both users.

RESULT ON PASS:

1. You are able to log in only with foo since bar account was deleted.

RESULT ON FAIL:

1. You still can log in with both users (foo, bar).

Implementation

Tags

• The bugs should be tagged based off the CD / Image they manifested themselves on, this tag should also indicate the ISO that it appeared on i.e. rc1-alternate CD.
• The Developers and members of the QA Team can tag the bugs if they wanted them added to the list.
• Advertise the tags being used by mailing ubuntu-devel and adding them to the wiki.

Criteria

• The criteria for identifying these bugs should be based on the scope of the impact of the bug, this will be done by:
  • Starting by looking at the importance of the bug report (Medium, High or Critical).

  • Look at the number of duplicate & subscribers (5 or more).

• Also a bug can be suggested as a candidate for the Fixed Bug Verification testing by someone that is not an Ubuntu Developer or a member of the QA Team, this will be done by adding a tag (fix-to-verify-candidate) to the report, later on the QA Team can approve or decline this candidate.

List

• Bughelper is going to be used for create a list of bugs that already have the fix-to-verify tag and need to be tested. And also will perform search based on the Criteria for identifying the bugs that need to be tagged fix-to-verify.
• Both list of bugs will be found at qa.ubuntu.com in two different static html pages.
• They should be ordered by when they have appeared with the most recent ones at the top of the list.

Infrastructure

• qa.ubuntu.com is necessary to have a better area for tracking these bugs.
• A feedback mechanism for testers to indicate that the bug verification was successful (bug is fixed) is needed.
  • This should happen in the feedback module of the qa tracker.
  • Regressions (bug comes back) should happen in launchpad - bug status should be flipped back to Triaged / Confirmed

Test/Demo Plan

Outstanding Issues

BoF agenda and discussion

CategorySpec