PerformingSRUVerification
Size: 2890
Comment:
|
← Revision 41 as of 2017-09-27 12:43:57 ⇥
Size: 8386
Comment: document ports.ubuntu.com. use a current release name in the documentation.
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
SRU Verification is the process of testing packages, updated to fix a bug, that exist in the -proposed repository. These packages need testing to ensure that the package continues to function as designed and that the bug is fixed. | <<Include(QATeam/Header)>> ||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;"><<TableOfContents>>|| Stable Release Update (SRU) Verification is the process of testing packages from the -proposed repository, that have been updated to fix bugs in a [[https://wiki.ubuntu.com/Releases|stable release]]. These packages need testing to ensure that the package continues to function as designed and that the bug is fixed. It is also important '''to ensure no regression has been introduced''' by the fix. More information about the process can be found at [[https://wiki.ubuntu.com/StableReleaseUpdates|Stable Release Update]] page. To find out how to enable -proposed look at the [[https://wiki.ubuntu.com/Testing/EnableProposed|Enable Proposed]] page. |
Line 5: | Line 11: |
Bugs needing SRU verification can be found by querying Launchpad for the bug tag [https://launchpad.net/ubuntu/+bugs?field.tag=verification-needed verification-needed] or by viewing the [http://people.ubuntu.com/~ubuntu-archive/pending-sru.html Pending Ubuntu SRUs]. The Pending Ubuntu SRUs are generated by parsing the changelogs of the packages in -proposed repository. By clicking on '''changelog bugs''' entry you will be taken to the Launchpad bug report (if it exists). Some bugs resolved by -proposed packages require specific hardware and these can be identified by the '''hw-specific''' tag in Launchpad or by the '''(hw)''' next to the bug number at the Pending Ubuntu SRUs page. | There are many ways to find bugs needing SRU verification: * by viewing the [[http://people.ubuntu.com/~ubuntu-archive/pending-sru.html|Pending Ubuntu SRUs]], * by asking aptitude from the command-line: "aptitude search ~i~Aproposed" * by querying Launchpad for a specific release (here lucid) and the bug tag [[https://launchpad.net/ubuntu/lucid/+bugs?field.status%3Alist=FIXCOMMITTED&field.tag=verification-needed|verification-needed]], * or by looking at bugs which the SRU verification team is [[https://bugs.launchpad.net/~sru-verification/+subscribedbugs|subscribed]]. The Pending Ubuntu SRUs are generated by parsing the changelogs of the packages in -proposed repository. By clicking on '''changelog bugs''' entry you will be taken to the Launchpad bug report (if it exists). If no changelog bug is displayed, clicking on the version of the package in the '-proposed' column will take you to the changelog entry for that version of the package in launchpad (the bug number may be displayed in the changelog but not in the report if the bug number doesn't respect a standard format) Some bugs resolved by -proposed packages require specific hardware and these can be identified by the '''hw-specific''' tag in Launchpad or by the '''(hw)''' next to the bug number at the Pending Ubuntu SRUs page. |
Line 9: | Line 24: |
The first step in identifying how to test is determing the release or releases of Ubuntu affected by the particular bug. This can be done by looking at the bug report and determining the release affected by the bug. (use a screenshot of bug 160176 to show what bugs targeted to a release look like). Alternatively, at the [http://people.ubuntu.com/~ubuntu-archive/pending-sru.html Pending Ubuntu SRUs] page there are sections for each release of Ubunut that is currently supported. | The first step in identifying how to test, is determining the release or releases of Ubuntu affected by the particular bug. This can be done by looking at the bug report and determining the release affected by the bug. |
Line 11: | Line 26: |
In addition to knowing the release or releases of Ubuntu affected you also need to have a detailed steps to recreate the bug. These can be found in the "TEST CASE" section at the end of the bug's description. | More than one release may be affected by the bug. In that case, the bug needs to be reproduced and the fix needs to be tested for each affected release. In the example below the Landspace bug affects the Jaunty, Karmic, Lucid and Maverick releases of Ubuntu and should be verified in each stable release (Jaunty, Karmic and Lucid) ||<tablestyle="float:center; font-size: 0.75em; background:#C2B8A1;margin: 0 0 1em 1em;" style="padding:0.5em;">[[https://wiki.ubuntu.com/QATeam/PerformingSRUVerification?action=AttachFile&do=get&target=bug_example_large.png|{{https://wiki.ubuntu.com/QATeam/PerformingSRUVerification?action=AttachFile&do=get&target=bug_example_small.png}}]]|| || Click to zoom || Alternatively, at the [[http://people.ubuntu.com/~ubuntu-archive/pending-sru.html|Pending Ubuntu SRUs]] page there are sections for each release of Ubuntu that is currently supported. In addition to knowing the release or releases of Ubuntu affected you also need to have detailed steps to recreate the bug. These can be found in the "TEST CASE" section at the end of the bug's description. [[https://bugs.edge.launchpad.net/ubuntu/karmic/+source/flashplugin-nonfree/+bug/429841|Flashplugin-nonfree bug 429841]] has an example of what the test case will look like. Test cases should be added to the description of the report in a standardised format: {{{ TEST CASE: 1. description of step 1 2. description of the step 2 3. ... VERIFICATION DONE What is the expected result at the end of the test. }}} Note that the verification-failed is either the subject of the bug report or a regression. Writing the test case is a mandatory step of the [[StableReleaseUpdates|SRU procedure]]. But sometimes, there is no such test case. If the test case is missing and the reproduction steps are not obvious, you can provide one if you can reproduce the issue described in the bug report or set the status to 'In progress', describe why you cannot reproduce and set the verification tag to 'verification-failed'. |
Line 17: | Line 55: |
1. Install all available updates | 1. Ensure that your system is up to date by installing all available updated packages from the -updates and -security repositories 1. Verify that you do not have the proposed package installed by checking the package version using 'dpkg -l PKGNAME | cat' |
Line 19: | Line 58: |
1. Update your '/etc/apt/sources.list' file to include the -proposed repository * what the line looks like for Gutsy 1. Install the updated package via 'sudo apt-get install package' or 'sudo apt-get install package=version-number' 1. If the package being tested is a "system" package reboot the system |
1. Modify your '/etc/apt/sources.list' file to include the -proposed repository as described in [[Testing/EnableProposed|how to enable -proposed]]. * deb http://archive.ubuntu.com/ubuntu/ xenial-proposed main restricted universe<<BR>>For verifying SRUs on non-x86 architectures, instead use: * deb http://ports.ubuntu.com/ubuntu-ports/ xenial-proposed main restricted universe 1. Execute 'sudo apt-get update' 1. Install the proposed package via 'sudo apt-get install PKGNAME' or 'sudo apt-get install PKGNAME=VERSION-NUM' 1. Verify that you installed the correct package version using 'dpkg -l PKGNAME | cat' 1. Reboot the system |
Line 25: | Line 67: |
1. Once you have finished testing the package from the -proposed repository you can disable -proposed. Failing to do so will lead to you updating any package available from the -proposed repository at the next apt update. Having a look at the patch may help to know exactly which part of the package is affected by the fix and what needs to be more specifically tested. One of the most important aspects of the testing procedure is to verify that the update does not introduce a regression. == Updating the bug report == <<Include(StableReleaseUpdates,,from="== Verification ==", to="If you want to help us")>> == Tagging the report == === SRU specific === <<Include(Bugs/Tags,,from='=== SRU Specific ===',to='=== X ')>> === Regression specific === <<Include(Bugs/Tags,,from='=== Regression specific ===',to='=== SRU Specific ===')>> |
|
Line 28: | Line 89: |
In the event that your current release of Ubuntu is not the same as the release of Ubuntu affected by the bug there are many ways for you to still perform the verification of the Stable Release Update without installing the affected release on your hardware. This can be done by using an emulator such as VMWare, virtual box, kvm or qemu. Depending on the nature of the bug report it may also be possible to use a chroot to perform the verification. | In the event that your current release of Ubuntu is not the same as the release of Ubuntu affected by the bug there are still many ways for you to perform the verification of the Stable Release Update without installing the affected release on your hardware. This can be done by using a [[https://help.ubuntu.com/community/VirtualMachines|Virtual Machine]] such as [[https://wiki.ubuntu.com/Testing/VirtualBox|Virtual Box]], [[https://wiki.ubuntu.com/KvmVirtManagerEtc|kvm]], [[https://help.ubuntu.com/community/Installation/QemuEmulator|qemu]] or VMware. Virtual Machines can not be used to reproduce hardware specific issues. Depending on the nature of the bug report it may also be possible to use a [[https://wiki.ubuntu.com/DebootstrapChroot|chroot]] to perform the verification. == Tips and tricks == * An useful tool to quickly check the versions of a package available in the active releases is `rmadison` from the package `devscripts`. * You can fetch the changelog of a package directly from the command line with `apt-get changelog PACKAGE_NAME` * The full publishing history of a package: https://launchpad.net/ubuntu/+source/PACKAGE_NAME/+publishinghistory * If you're wondering where is the uploaded package you should check [[ https://launchpad.net/ubuntu/oneiric/+queue|the state of the upload queue]] == Links == * [[StableReleaseUpdates|Stable Release Update]] procedure * [[http://people.ubuntu.com/~ubuntu-archive/pending-sru.html|Pending Ubuntu SRUs]] * List of bugs which the SRU verification team is [[https://bugs.launchpad.net/~sru-verification/+subscribedbugs|subscribed]] * How to [[Testing/EnableProposed|Enable Proposed]] * How to triage [[Bugs/HowToTriage|guide]] |
Stable Release Update (SRU) Verification is the process of testing packages from the -proposed repository, that have been updated to fix bugs in a stable release. These packages need testing to ensure that the package continues to function as designed and that the bug is fixed. It is also important to ensure no regression has been introduced by the fix.
More information about the process can be found at Stable Release Update page. To find out how to enable -proposed look at the Enable Proposed page.
How to find bugs needing verification
There are many ways to find bugs needing SRU verification:
by viewing the Pending Ubuntu SRUs,
- by asking aptitude from the command-line: "aptitude search ~i~Aproposed"
by querying Launchpad for a specific release (here lucid) and the bug tag verification-needed,
or by looking at bugs which the SRU verification team is subscribed.
The Pending Ubuntu SRUs are generated by parsing the changelogs of the packages in -proposed repository. By clicking on changelog bugs entry you will be taken to the Launchpad bug report (if it exists). If no changelog bug is displayed, clicking on the version of the package in the '-proposed' column will take you to the changelog entry for that version of the package in launchpad (the bug number may be displayed in the changelog but not in the report if the bug number doesn't respect a standard format)
Some bugs resolved by -proposed packages require specific hardware and these can be identified by the hw-specific tag in Launchpad or by the (hw) next to the bug number at the Pending Ubuntu SRUs page.
Identifying how to test
The first step in identifying how to test, is determining the release or releases of Ubuntu affected by the particular bug. This can be done by looking at the bug report and determining the release affected by the bug.
More than one release may be affected by the bug. In that case, the bug needs to be reproduced and the fix needs to be tested for each affected release.
In the example below the Landspace bug affects the Jaunty, Karmic, Lucid and Maverick releases of Ubuntu and should be verified in each stable release (Jaunty, Karmic and Lucid)
Alternatively, at the Pending Ubuntu SRUs page there are sections for each release of Ubuntu that is currently supported.
In addition to knowing the release or releases of Ubuntu affected you also need to have detailed steps to recreate the bug. These can be found in the "TEST CASE" section at the end of the bug's description. Flashplugin-nonfree bug 429841 has an example of what the test case will look like.
Test cases should be added to the description of the report in a standardised format:
TEST CASE: 1. description of step 1 2. description of the step 2 3. ... VERIFICATION DONE What is the expected result at the end of the test.
Note that the verification-failed is either the subject of the bug report or a regression.
Writing the test case is a mandatory step of the SRU procedure. But sometimes, there is no such test case. If the test case is missing and the reproduction steps are not obvious, you can provide one if you can reproduce the issue described in the bug report or set the status to 'In progress', describe why you cannot reproduce and set the verification tag to 'verification-failed'.
How to perform the test
After booting into the affected release of Ubuntu the following steps should be taken:
- Ensure that your system is up to date by installing all available updated packages from the -updates and -security repositories
- Verify that you do not have the proposed package installed by checking the package version using 'dpkg -l PKGNAME | cat'
- Recreate the bug using the steps identified in the "TEST CASE"
Modify your '/etc/apt/sources.list' file to include the -proposed repository as described in how to enable -proposed.
deb http://archive.ubuntu.com/ubuntu/ xenial-proposed main restricted universe
For verifying SRUs on non-x86 architectures, instead use:deb http://ports.ubuntu.com/ubuntu-ports/ xenial-proposed main restricted universe
- Execute 'sudo apt-get update'
- Install the proposed package via 'sudo apt-get install PKGNAME' or 'sudo apt-get install PKGNAME=VERSION-NUM'
- Verify that you installed the correct package version using 'dpkg -l PKGNAME | cat'
- Reboot the system
- Try to recreate the bug using the steps identified in the "TEST CASE"
- Use the software installed by the package in common ways
- Once you have finished testing the package from the -proposed repository you can disable -proposed. Failing to do so will lead to you updating any package available from the -proposed repository at the next apt update.
Having a look at the patch may help to know exactly which part of the package is affected by the fix and what needs to be more specifically tested.
One of the most important aspects of the testing procedure is to verify that the update does not introduce a regression.
Updating the bug report
Verification must be done in a software environment as close as is feasible to that which will exist after the package is copied to *-updates. Generally this will be with a system that is up to date from *-release, *-security, and *-updates, but not with other packages from *-proposed (except other packages built from the affected source package - they must be updated if generally installed) or *-backports. While not ideal it is also possible for the uploader of the fix to perform the verification of the package in *-proposed, however it must still be done in a software environment as close as is feasible to that which will exist after the package is copied to *-updates.
If they discover that your fix is insufficient, or the test case is not sufficient to reproduce the bug, they will:
Set the bug task to Status: In Progress
- Describe why the fix was insufficient or broken in a comment to the bug report.
Modify the verification-needed-$RELEASE tag to a verification-failed-$RELEASE tag on the bug report where $RELEASE is the release name which you tested.
The SRU verification team may also discover that your fix is good. They will:
Modify the 'verification-needed-$RELEASE' tag to a 'verification-done-$RELEASE' tag on the bug report where $RELEASE is the release name of the upload you have tested e.g. verification-done-precise. The tag global verification-needed should be left on the bug until all release tasks have been verified.
- Describe the general steps taken to verify the package (and any special difficulties) in a comment on the bug.
Verification feedback from bug reporters and subscribers is greatly appreciated, too, especially if the update is hardware specific. In this case we consider an update as verified if it has at least two positive, no negative testimonials in the bug report, and the verification team just checks whether the new version still works for the main use cases (to check for major regressions).
If you encounter a regression in a package that has been uploaded to *-proposed, please:
- File a bug report against the package, describing the nature of the regression you have encountered, including any special steps needed to reproduce the regression.
Mark this bug with the tag regression-proposed
Ask a bug supervisor to target the bug to the appropriate Ubuntu releases.
- Follow up on the SRU bug report referenced from the package changelog, pointing to the new bug. If there is more than one bug in the SRU changelog, follow up to the bug that is most closely related to the regression.
Set the verification-failed-$RELEASE tag on the corresponding SRU bug report where $RELEASE is the release name of the upload you tested.
Tagging the report
SRU specific
See StableReleaseUpdates for more information.
Tag |
Use case |
A Stable Release Update bug with a package in -proposed that has been confirmed to fix the bug. |
|
A Stable Release Update bug with a package in -proposed that has been verified to not fix the bug. |
|
A Stable Release Update bug with a package in -proposed needing testing. |
|
block-proposed-<series> |
A bug that should be held back being released into the updates pocket for the corresponding stable series. See StableReleaseUpdates#Staging_an_upload. |
Regression specific
See the regression tracker for a list of these bugs and QATeam/RegressionTracking for more information.
Tag |
Use case |
A bug in a release that was not present in a previous release. Should be used together with a separate tag for the release the regression was found in. This also applies to a development release where an update introduces a regression prior to its official release. |
|
A bug in a stable release that was introduced by a package from -updates. This would not apply to a development release where a regression was introduced prior to its official release. |
|
A bug in a stable release of Ubuntu that was found when testing a package from -proposed. |
|
An apport crash bug report that was identified by the retracer as having the same characteristics as a fixed crash report. |
|
This is for when a bisect is requested or required to fix the bug. |
|
The reporter, developer, or triager is in the process of performing a bisection. |
|
Add when a bisect has been completed and the offending commit(s) identified. |
Release Specific
Tag |
Use case |
A bug tracking a cause of a package not being migrated to the release pocket. This tag causes the bug to appear on the update excuses report. |
|
A bug that should be held back from migrating into the release pocket |
Ways to test using virtual machines
In the event that your current release of Ubuntu is not the same as the release of Ubuntu affected by the bug there are still many ways for you to perform the verification of the Stable Release Update without installing the affected release on your hardware.
This can be done by using a Virtual Machine such as Virtual Box, kvm, qemu or VMware.
Virtual Machines can not be used to reproduce hardware specific issues.
Depending on the nature of the bug report it may also be possible to use a chroot to perform the verification.
Tips and tricks
An useful tool to quickly check the versions of a package available in the active releases is rmadison from the package devscripts.
You can fetch the changelog of a package directly from the command line with apt-get changelog PACKAGE_NAME
The full publishing history of a package: https://launchpad.net/ubuntu/+source/PACKAGE_NAME/+publishinghistory
If you're wondering where is the uploaded package you should check the state of the upload queue
Links
Stable Release Update procedure
List of bugs which the SRU verification team is subscribed
How to Enable Proposed
How to triage guide
QATeam/PerformingSRUVerification (last edited 2017-09-27 12:43:57 by vorlon)