PerformingSRUVerification

Stable Release Update (SRU) Verification is the process of testing packages from the -proposed repository, that have been updated to fix bugs in a stable release. These packages need testing to ensure that the package continues to function as designed and that the bug is fixed. It is also important to ensure no regression has been introduced by the fix.

More information about the process can be found at Stable Release Update page. To find out how to enable -proposed look at the Enable Proposed page.

How to find bugs needing verification

There are many ways to find bugs needing SRU verification:

  • by viewing the Pending Ubuntu SRUs,

  • by asking aptitude from the command-line: "aptitude search ~i~Aproposed"
  • by querying Launchpad for a specific release (here lucid) and the bug tag verification-needed,

  • or by looking at bugs which the SRU verification team is subscribed.

The Pending Ubuntu SRUs are generated by parsing the changelogs of the packages in -proposed repository. By clicking on changelog bugs entry you will be taken to the Launchpad bug report (if it exists). If no changelog bug is displayed, clicking on the version of the package in the '-proposed' column will take you to the changelog entry for that version of the package in launchpad (the bug number may be displayed in the changelog but not in the report if the bug number doesn't respect a standard format)

Some bugs resolved by -proposed packages require specific hardware and these can be identified by the hw-specific tag in Launchpad or by the (hw) next to the bug number at the Pending Ubuntu SRUs page.

Identifying how to test

The first step in identifying how to test, is determining the release or releases of Ubuntu affected by the particular bug. This can be done by looking at the bug report and determining the release affected by the bug.

More than one release may be affected by the bug. In that case, the bug needs to be reproduced and the fix needs to be tested for each affected release.

In the example below the Landspace bug affects the Jaunty, Karmic, Lucid and Maverick releases of Ubuntu and should be verified in each stable release (Jaunty, Karmic and Lucid)

  • https://wiki.ubuntu.com/QATeam/PerformingSRUVerification?action=AttachFile&do=get&target=bug_example_large.png

    Click to zoom

Alternatively, at the Pending Ubuntu SRUs page there are sections for each release of Ubuntu that is currently supported.

In addition to knowing the release or releases of Ubuntu affected you also need to have detailed steps to recreate the bug. These can be found in the "TEST CASE" section at the end of the bug's description. Flashplugin-nonfree bug 429841 has an example of what the test case will look like.

Test cases should be added to the description of the report in a standardised format:

TEST CASE:
1. description of step 1
2. description of the step 2
3. ...

VERIFICATION DONE
What is the expected result at the end of the test.

Note that the verification-failed is either the subject of the bug report or a regression.

Writing the test case is a mandatory step of the SRU procedure. But sometimes, there is no such test case. If the test case is missing and the reproduction steps are not obvious, you can provide one if you can reproduce the issue described in the bug report or set the status to 'In progress', describe why you cannot reproduce and set the verification tag to 'verification-failed'.

How to perform the test

After booting into the affected release of Ubuntu the following steps should be taken:

  1. Ensure that your system is up to date by installing all available updated packages from the -updates and -security repositories
  2. Verify that you do not have the proposed package installed by checking the package version using 'dpkg -l PKGNAME | cat'
  3. Recreate the bug using the steps identified in the "TEST CASE"
  4. Modify your '/etc/apt/sources.list' file to include the -proposed repository as described in how to enable -proposed.

  5. Execute 'sudo apt-get update'
  6. Install the proposed package via 'sudo apt-get install PKGNAME' or 'sudo apt-get install PKGNAME=VERSION-NUM'
  7. Verify that you installed the correct package version using 'dpkg -l PKGNAME | cat'
  8. Reboot the system
  9. Try to recreate the bug using the steps identified in the "TEST CASE"
  10. Use the software installed by the package in common ways
  11. Once you have finished testing the package from the -proposed repository you can disable -proposed. Failing to do so will lead to you updating any package available from the -proposed repository at the next apt update.

Having a look at the patch may help to know exactly which part of the package is affected by the fix and what needs to be more specifically tested.

One of the most important aspects of the testing procedure is to verify that the update does not introduce a regression.

Updating the bug report

Verification must be done in a software environment as close as is feasible to that which will exist after the package is copied to *-updates. Generally this will be with a system that is up to date from *-release, *-security, and *-updates, but not with other packages from *-proposed (except other packages built from the affected source package - they must be updated if generally installed) or *-backports. While not ideal it is also possible for the uploader of the fix to perform the verification of the package in *-proposed, however it must still be done in a software environment as close as is feasible to that which will exist after the package is copied to *-updates.

If they discover that your fix is insufficient, or the test case is not sufficient to reproduce the bug, they will:

  1. Set the bug task to Status: In Progress

  2. Describe why the fix was insufficient or broken in a comment to the bug report.
  3. Modify the verification-needed-$RELEASE tag to a verification-failed-$RELEASE tag on the bug report where $RELEASE is the release name which you tested.

The SRU verification team may also discover that your fix is good. They will:

  1. Modify the 'verification-needed-$RELEASE' tag to a 'verification-done-$RELEASE' tag on the bug report where $RELEASE is the release name of the upload you have tested e.g. verification-done-precise. The tag global verification-needed should be left on the bug until all release tasks have been verified.

  2. Describe the general steps taken to verify the package (and any special difficulties) in a comment on the bug.

Verification feedback from bug reporters and subscribers is greatly appreciated, too, especially if the update is hardware specific. In this case we consider an update as verified if it has at least two positive, no negative testimonials in the bug report, and the verification team just checks whether the new version still works for the main use cases (to check for major regressions).

If you encounter a regression in a package that has been uploaded to *-proposed, please:

  1. File a bug report against the package, describing the nature of the regression you have encountered, including any special steps needed to reproduce the regression.
  2. Mark this bug with the tag regression-proposed

  3. Ask a bug supervisor to target the bug to the appropriate Ubuntu releases.

  4. Follow up on the SRU bug report referenced from the package changelog, pointing to the new bug. If there is more than one bug in the SRU changelog, follow up to the bug that is most closely related to the regression.
  5. Set the verification-failed-$RELEASE tag on the corresponding SRU bug report where $RELEASE is the release name of the upload you tested.

Tagging the report

SRU specific

See StableReleaseUpdates for more information.

Tag

Use case

verification-done

A Stable Release Update bug with a package in -proposed that has been confirmed to fix the bug.

verification-failed

A Stable Release Update bug with a package in -proposed that has been verified to not fix the bug.

verification-needed

A Stable Release Update bug with a package in -proposed needing testing.

block-proposed-<series>

A bug that should be held back being released into the updates pocket for the corresponding stable series. See StableReleaseUpdates#Staging_an_upload.

Regression specific

See the regression tracker for a list of these bugs and QATeam/RegressionTracking for more information.

Tag

Use case

regression-release

A bug in a release that was not present in a previous release. Should be used together with a separate tag for the release the regression was found in. This also applies to a development release where an update introduces a regression prior to its official release.

regression-update

A bug in a stable release that was introduced by a package from -updates. This would not apply to a development release where a regression was introduced prior to its official release.

regression-proposed

A bug in a stable release of Ubuntu that was found when testing a package from -proposed.

regression-retracer

An apport crash bug report that was identified by the retracer as having the same characteristics as a fixed crash report.

needs-bisect

This is for when a bisect is requested or required to fix the bug.

performing-bisect

The reporter, developer, or triager is in the process of performing a bisection.

bisect-done

Add when a bisect has been completed and the offending commit(s) identified.

Release Specific

Tag

Use case

update-excuse

A bug tracking a cause of a package not being migrated to the release pocket. This tag causes the bug to appear on the update excuses report.

block-proposed

A bug that should be held back from migrating into the release pocket

Ways to test using virtual machines

In the event that your current release of Ubuntu is not the same as the release of Ubuntu affected by the bug there are still many ways for you to perform the verification of the Stable Release Update without installing the affected release on your hardware.

This can be done by using a Virtual Machine such as Virtual Box, kvm, qemu or VMware.

Virtual Machines can not be used to reproduce hardware specific issues.

Depending on the nature of the bug report it may also be possible to use a chroot to perform the verification.

Tips and tricks

QATeam/PerformingSRUVerification (last edited 2017-09-27 12:43:57 by vorlon)