Ubuntu Wiki

Verification must be done in a software environment as close as is feasible to that which will exist after the package is copied to *-updates. Generally this will be with a system that is up to date from *-release, *-security, and *-updates, but not with other packages from *-proposed (except other packages built from the affected source package - they must be updated if generally installed) or *-backports. While not ideal it is also possible for the uploader of the fix to perform the verification of the package in *-proposed, however it must still be done in a software environment as close as is feasible to that which will exist after the package is copied to *-updates.

If they discover that your fix is insufficient, or the test case is not sufficient to reproduce the bug, they will:

1. Set the bug task to Status: In Progress

2. Describe why the fix was insufficient or broken in a comment to the bug report.

3. Modify the verification-needed-$RELEASE tag to a verification-failed-$RELEASE tag on the bug report where $RELEASE is the release name which you tested.

The SRU verification team may also discover that your fix is good. They will:

1. Modify the 'verification-needed-$RELEASE' tag to a 'verification-done-$RELEASE' tag on the bug report where $RELEASE is the release name of the upload you have tested e.g. verification-done-precise. The tag global verification-needed should be left on the bug until all release tasks have been verified.

2. Describe the general steps taken to verify the package (and any special difficulties) in a comment on the bug.

Verification feedback from bug reporters and subscribers is greatly appreciated, too, especially if the update is hardware specific. In this case we consider an update as verified if it has at least two positive, no negative testimonials in the bug report, and the verification team just checks whether the new version still works for the main use cases (to check for major regressions).

If you encounter a regression in a package that has been uploaded to *-proposed, please:

1. File a bug report against the package, describing the nature of the regression you have encountered, including any special steps needed to reproduce the regression.

2. Mark this bug with the tag regression-proposed

3. Ask a bug supervisor to target the bug to the appropriate Ubuntu releases.

4. Follow up on the SRU bug report referenced from the package changelog, pointing to the new bug. If there is more than one bug in the SRU changelog, follow up to the bug that is most closely related to the regression.

5. Set the verification-failed-$RELEASE tag on the corresponding SRU bug report where $RELEASE is the release name of the upload you tested.

If you want to help us to verify Stable Release Updates then read how to perform a Stable Release Update verification

Verification Notes

1. There is a standing agenda item for the SRU & LTS meeting to make sure all stakeholders are able to raise issues as early as possible.

2. Ensure all critical and high importance bugs are verified in a timely manner. If not, the SRU QA Engineer will perform the testing.

3. The SRU QA Engineer will specifically ask at the SRU & LTS meeting if there are specific bugs that need verification that aren't being done by the bug reporter. If necessary, a QA team member will do the verification. If not able (e.g. lack of specific HW), will do more calls for testing and nag the bug reporter again.

4. If necessary, the QA team will set up separate SRU verification program, for big packages like eglibc, python, X.

Autopkgtest Regressions

Packages accepted into -proposed as per the SRU process automatically trigger related autopkgtests, similarly to how it happens for the development series. Once those tests are finished, the pending SRU page provides links to any failures that have been noticed for the selected upload. The responsibility of the uploader (and/or the person performing update verification) is to make sure the upload does not cause any regressions - both in manual and automated testing.

In the case where an SRU upload triggers an autopkgtest regression, the target package will not be released into -updates until the failure is resolved. There are a few ways that can happen:

• If the reported autopkgtest regression is a real regression caused by the upload, the update should be considered verification-failed and the package should be re-uploaded with the regression fixed. Otherwise the update will be removed from -proposed as per the usual procedures.

• If the reported autopkgtest regression is not a real regression or not a regression caused by the proposed update (but instead broken by some other dependency), the analysis of this has to be documented for the SRU team. The generally recommended way is commenting on one of the SRU bugs for the upload. Once the rationale is submitted and approved/validated by an SRU member, the SRU team will add a badtest or reset-test hint for the broken package and release the update as per usual procedures (once validation and aging is complete). Alternatively, the uploader/verifier can modify the hints and provide an MP in the bug along with the rationale. Useful input here can be re-running the failing test against only the release/updates pocket, as documented in the ProposedMigration wiki page.

• If the reported autopkgtest regression is the result of a flaky test, the uploader can try re-running the test to see if it is indeed just a transient issue. If the issue still persists but the analysis clearly shows that it is not a real regression, a rationale for that should be provided in one of the SRU bugs.

It is important to remember that firstly it is the uploader's responsibility to make sure the package is in a releasable state and that all the autopkgtests triggered by the upload are either passing or badtested. Of course, it is not the uploader's responsibility to provide the hints for badtests themselves, but it is it's responsibility to perform the analysis and verification of each listed regression.

Expected resolution for reported autopkgtest failures

If an SRU did not cause a regression, the update is intended to land but autopkgtest regressions are still found, then the possible resolutions for this situation are as follows:

• An SRU must not be released if outstanding autopkgtest regressions are reported on the Pending SRU Report.

• Ideally regressions can be fixed with new uploads to the proposed pockets as required. Fixes for autopkgtests are generally always acceptable. However uploads providing only test fixes will generally be staged using block-proposed-<series> (in which case they need a bug reference).

• A regression not caused by the SRU may be badtest or reset-tested away (doesn't matter which for SRUs).

Removal of updates

If a bug fixed by an update does not get any testing/verification feedback for 90 days an automated call for testing comment will be made on the bug report. In the event that there is still no testing after an additional 15 days, that's a total of 105 days without any testing, the Stable Release Managers will remove the package from -proposed and usually close the bug task as "Won't Fix", due to lack of interest. Removal will happen immediately if a package update in -proposed is found to introduce a nontrivial regression.

Regressions