Ubuntu Wiki

Once an Ubuntu release has been completed and published, updates for it are only released under certain circumstances, and must follow a special procedure called a "stable release update" or SRU.

There is an automatically generated list of packages which are currently undergoing this process.

Did you notice a regression in a package which went to -updates? Please report this using these steps.

1. Why

In contrast to pre-release versions, official releases of Ubuntu are subject to much wider use, and by a different demographic of users. During development, changes to the distribution primarily affect developers, early adopters and other advanced users, all of whom have elected to use pre-release software at their own risk.

Users of the official release, in contrast, expect a high degree of stability. They use their Ubuntu system for their day-to-day work, and problems they experience with it can be extremely disruptive. Many of them are less experienced with Ubuntu and with Linux, and expect a reliable system which does not require their intervention.

Stable release updates are automatically recommended to a very large number of users, and so it is critically important to treat them with great caution. Therefore, when updates are proposed, they must be accompanied by a strong rationale and present a low risk of regressions.

• "It's just a one-line change!"

Even the simplest of changes can cause unexpected regressions due to lurking problems:

• In bug 81125, the upgrade regression had nothing to do with the content of the change that triggered it: any user who had installed the libpthread20 package would encounter a problem the next time libc6 was upgraded.

• In bug 309674, the failure was a misbuild due to timestamp skew in the build process. The underlying problem existed in the source package in the original release, but would only manifest in a small percentage of builds.

• In bug 559822, a C++ library (wxwidgets2.8) was uploaded with no code changes. Due to an underlying toolchain change/bug, this caused an ABI change, causing a lot of unrelated packages to break (see bug 610975)

We never assume that any change, no matter how obvious, is completely free of regression risk.

In line with this, the requirements for stable updates are not necessarily the same as those in the development release. When preparing future releases, one of our goals is to construct the most elegant and maintainable system possible, and this often involves fundamental improvements to the system's architecture, rearranging packages to avoid bundled copies of other software so that we only have to maintain it in one place, and so on. However, once we have completed a release, the priority is normally to minimise risk caused by changes not explicitly required to fix qualifying bugs, and this tends to be well-correlated with minimising the size of those changes. As such, the same bug may need to be fixed in different ways in stable and development releases.

2. When

2.1. High-impact bugs

Stable release updates will, in general, only be issued in order to fix high-impact bugs. Examples of such bugs include:

• Bugs which may, under realistic circumstances, directly cause a security vulnerability. These are done by the security team and are documented at SecurityTeam/UpdateProcedures.

• Bugs which represent severe regressions from the previous release of Ubuntu. This includes packages which are totally unusable, like being uninstallable or crashing on startup.

• Bugs which may, under realistic circumstances, directly cause a loss of user data

• Updates that need to be applied to Ubuntu packages to adjust to changes in the environment, server protocols, web services, and similar, i. e. where the current version just ceases to work. Examples:
  • app-install-data-commercial is a package index which regularly needs to be adjusted to changes in the commercial package archive.

  • clamav needs regular updates to latest virus signatures

  • tor needs a newer version to still work with the current Tor network.
  • A library for a web service needs to be updated for changes to the web server API.

2.2. Other safe cases

In the following cases a stable release update is also applicable as they have a low potential for regressing existing installations but a high potential for improving the user experience, particularly for Long Term Support releases:

• Bugs which do not fit under above categories, but (1) have an obviously safe patch and (2) affect an application rather than critical infrastructure packages (like X.org or the kernel).
• For Long Term Support releases we regularly want to enable new hardware. Such changes are appropriate provided that we can ensure not to affect upgrades on existing hardware. For example, modaliases of newly introduced drivers must not overlap with previously shipped drivers. This also includes updating hardware description data such as udev's keymaps, media-player-info, mobile broadband vendors, or PCI vendor/product list updates.
• For Long Term Support releases we sometimes want to introduce new features. They must not change the behaviour on existing installations (e. g. entirely new packages are usually fine). If existing software needs to be modified to make use of the new feature, it must be demonstrated that these changes are unintrusive, have a minimal regression potential, and have been tested properly. To avoid regressions on upgrade, any such feature must then also be added to any newer supported Ubuntu release. Once a new feature/package has been introduced, subsequent changes to it are subject to the usual requirements of SRUs to avoid regressions.
• New versions of commercial software in the Canonical partner archive.

• FTBFS(Fails To Build From Source) can also be considered. Please note that in main the release process ensures that there are no binaries which are not built from a current source. Usually those bugs should only be SRUed in conjunction with another bug fix.

For new upstream versions of packages which provide new features, but don't fix critical bugs, a backport should be requested instead.

2.3. New upstream microreleases

In some cases, when upstream fixes bugs, they do a new microrelease instead of just sending patches. If all of the changes are appropriate for an SRU by the criteria above, then it is acceptable (and usually easier) to just upload the complete new upstream microrelease instead of backporting the individual patches. Note that some noise introduced by autoreconf is okay, but making structural changes to the build system (such as introducing new library dependencies) is generally not.

For upstreams who have

• a reliable and credible test suite for assuring the quality of every commit or release,
• the tests are covering both functionality and API/ABI stability,
• the tests run during package build to cover all architectures,

• the package has an autopkgtest to run the tests in an Ubuntu environment against the actual binary packages,

it is also acceptable to upload new microreleases with many bug fixes without individual Launchpad bugs for each of them (~ubuntu-sru will make the final decision). The upstream QA process must be documented/demonstrated and linked from the SRU tracking bug. In other cases where such upstream automatic testing is not available, exceptions must still be approved by at least one member of the Ubuntu Technical Board.

3. Procedure

1. Check that the bug is fixed in the current development release, and that its bug task is "Fix Released". Equivalently for new upstream releases, this (or a a newer) release must be in the development release. It is, in general, not appropriate to release updates for stable systems without first testing them in the current development branch.
2. Ensure that the bug report for this issue is public. If the bug has been reported privately and cannot be published, you must first create a separate public bug report in launchpad and copy over as much information as can be published.

3. Update the bug report description and make sure it contains the following information:

   1. An explanation of the bug on users and justification for backporting the fix to the stable release. This may be preceded with an [Impact] header, but this is not required. In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug.

   2. A [Test Case] section with detailed instructions how to reproduce the bug. These should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem.

   3. A [Regression Potential] section with a discussion of how regressions are most likely to manifest as a result of this change. It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what could happen in the event of a regression. In the case of new upstream releases, link to the upstream QA procedure/documentation. This both shows the SRU team that the risks have been considered, and provides guidance to testers in regression-testing the SRU.

4. Ask the Ubuntu bug control team to nominate the bug for the appropriate Ubuntu release(s)/series (e. g. the current LTS and latest stable release). You can ask on IRC (Freenode) in #ubuntu-bugs, or by emailing them at the address found on the linked page.

5. Upload the fixed package to release-proposed with the patch in the bug report, a detailed and user-readable changelog, and no other unrelated changes. If you can't upload to the archive yourself, get a sponsor, attach a debdiff to the bug and subscribe ubuntu-sponsors, as usual. There is no need to wait before uploading. After upload, the bug status should be changed to In Progress, once accepted into release-proposed, the status will be automatically changed to Fix Committed. Also, make sure that:

   1. The version number does not conflict with any later and future version in other Ubuntu releases (the security policy document has a well-working scheme which can be used for SRUs.)

   2. There is a reference to the SRU bug number in the changelog, using the 'LP: #NNNNNN' convention. Only public bugs should be referenced in the changelog.

6. The ~ubuntu-sru team will review and approve then the archive admins will accept your upload. You can then test the actual binaries in the Ubuntu archive yourself and follow up in the bug report regarding your verification of the bug. The SRU team will evaluate the testing feedback and they will move the package into -updates after it has passed a minimum aging period of 7 days.

7. Subscribe yourself to bugmail for the package in Launchpad, if you haven't done so already, and monitor Launchpad for bug reports relating to the update for at least one week.

   Any regression must always be documented in a bug report, which must be Importance: critical once the regression has been confirmed.

3.1. SRU Bug Template

[Impact] 

 * An explanation of the effects of the bug on users and

 * justification for backporting the fix to the stable release.

 * In addition, it is helpful, but not required, to include an
   explanation of how the upload fixes this bug.

[Test Case]

 * detailed instructions how to reproduce the bug

 * these should allow someone who is not familiar with the affected
   package to reproduce the bug and verify that the updated package fixes
   the problem.

[Regression Potential] 

 * discussion of how regressions are most likely to manifest as a result of this change. 

 * It is assumed that any SRU candidate patch is well-tested before
   upload and has a low overall risk of regression, but it's important
   to make the effort to think about what ''could'' happen in the
   event of a regression.

 * This both shows the SRU team that the risks have been considered,
   and provides guidance to testers in regression-testing the SRU.

[Other Info]
 
 * Anything else you think is useful to include
 * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
 * and address these questions in advance

4. Publishing

The Stable Release Updates team regularly checks for SRUs that have succesfully completed verification (all bugs are marked verification-done for the series) and releases those to the -updates pocket. Having said that, if the there is a priority SRU waiting in the unapproved queue for release to -proposed, or needing release to -updates, feel free to contact an SRU vanguard. Vanguards from the SRU team can be found in #ubuntu-release on the following schedule:

5. Phasing

Once an update is released to -updates, the update is then phased so that the update is gradually made available to expanding subsets of Ubuntu users. This process allows us to automatically monitor for regressions and halt the update process if any are found. Complete details about the process can be found in a blog post by Brian Murray.

The Phased-Update-Percentage is initially set to 10%, and a job is run (every 6 hours) that checks for regressions and if none are found the phased update percentage will be incremented by 10%. So an update will become fully phased after 54 hours or about 2 days. In the event that a regression is detected the Phased-Update-Precentage will be set to 0% thereby causing supported package managers not to install the update.

The progress of phased updates is visible in a report which is updated by the same job that does the phasing.

6. Fixing several bugs in one upload

Please avoid creating meta-bugs like "Please SRU this". They just create redundancy and are opaque to original bug reporters, whose feedback is valuable for verification, so these bugs will generally be invalidated by the SRU team. Just prepare all fixed bugs as described above, and either

• attach the complete patch/debdiff to one bug and point to it in the other bugs ("debdiff which fixes this is attached to bug #xxxxxx")

or

• attach individual patches to the corresponding bug reports. If you have the fixes in bzr, it is even easier and more convenient to give a pointer to the fix ("fixed in http://bazaar.launchpad.net/.../revision/12") when fixing the bug in trunk.

7. Verification

The SRU verification team will regularly check open bugs with the verification-needed tag and test proposed updates on different hardware to check for inadvertent side effects. Verification must be done in a software environment as close as is feasible to that which will exist after the package is copied to *-updates. Generally this will be with a system that is up to date from *-release, *-security, and *-updates, but not with other packages from *-proposed (except other packages built from the affected source package - they must be updated if generally installed) or *-backports.

If they discover that your fix is insufficient, or the test case is not sufficient to reproduce the bug, they will:

1. Set the bug task to Status: In Progress

2. Describe why the fix was rejected in a comment to the bug report.

3. Modify the verification-needed tag to a verification-failed tag on the bug report.

The SRU verification team may also discover that your fix is good. They will:

1. Modify the verification-needed tag to a verification-done tag on the bug report. (In the event that an SRU bug report has tasks for mulitple releases a tag in the form of 'verification-done-$RELEASE' e.g. verification-done-precise shall be added to the bug report. The tag verification-needed should be left on the bug until all release tasks have been verified.)

2. Describe the general steps taken to verify the package, and any special difficulties.

While not ideal it is also possible for the uploader of the fix to perform the verification of the package in *-proposed, however it must still be done in a software environment as close as is feasible to that which will exist after the package is copied to *-updates.

Verification feedback from bug reporters and subscribers is greatly appreciated, too, especially if the update is hardware specific. In this case we consider an update as verified if it has at least two positive, no negative testimonials in the bug report, and the verification team just checks whether the new version still works for the main use cases (to check for major regressions).

If you encounter a regression in a package that has been uploaded to *-proposed, please:

1. File a bug report against the package, describing the nature of the regression you have encountered, including any special steps needed to reproduce the regression.

2. Mark this bug with the tag regression-proposed

3. Ask a bug supervisor to target the bug to the appropriate Ubuntu releases.

4. Follow up to the SRU bug report referenced from the package changelog, pointing to the new bug. If there is more than one bug in the SRU changelog, follow up to the bug that is most closely related to the regression.

5. Set the verification-failed tag on the corresponding SRU bug report.

If you want to help us to verify Stable Release Updates then read how to perform a Stable Release Update verification

Verification Notes

1. There is a standing agenda item for the SRU & LTS meeting to make sure all stakeholders are able to raise issues as early as possible.

2. Ensure all critical and high importance bugs are verified in a timely manner. If not, the SRU QA Engineer will perform the testing.

3. The SRU QA Engineer will specifically ask at the SRU & LTS meeting if there are specific bugs that need verification that aren't being done by the bug reporter. If necessary, a QA team member will do the verification. If not able (e.g. lack of specific HW), will do more calls for testing and nag the bug reporter again.

4. If necessary, the QA team will set up separate SRU verification program, for big packages like eglibc, python, X.

8. Removal of updates

If a bug fixed by an update does not get any testing/verification feedback for 90 days an automated call for testing comment will be made on the bug report. In the event that there is still no testing after an additional 15 days, that's a total of 105 days without any testing, the Stable Release Managers will remove the package from -proposed and usually close the bug task as "Won't Fix", due to lack of interest. Removal will happen immediately if a package update in -proposed is found to introduce a nontrivial regression.

9. Regressions

If a package update introduces a regression which already made it through the verification process to -updates, please immediately join #ubuntu-devel on Freenode, send the message "!regression-alert" on its own, and then describe the problem. Eg:

16:33 < you> !regression-alert
16:33 < ubottu> ... <automated response> ...
16:33 < you> bug #... <your explanation>

Please include the package name and a relevant bug number if possible. If you cannot join IRC, please follow up on one of the relevant bugs (see changelog).

If the regression only applies to the package in -proposed, please follow up to the bug with a detailed explanation, and tag the bug with regression-proposed.

9.1. Testing for Regressions

To minimise the risk of regressions being introduced via a SRU, testing will be perform by Canonical on each proposed kernel.

Depth Regression testing will be performed by the Ubuntu Platform QA team on minimal set of HW that represents the different flavours of Ubuntu Editions and Architectures. This activity will focus on verifying that hw-independent regressions have not been introduced.

Breadth hardware testing will be performed by the HW Certification team on release-certified HW. The test will verify that the proposed kernel can be successfully installed on the latest (point) release, network access is functional, and no other functionality is missing that will enable Update Manager to work correctly.

10. Documentation for Special Cases

The Technical Board resolution on Landscape provides a general rationale for the types of special cases that may be approved here in future.

10.1. Kernel

Because of the way updates to the kernel work, it will follow a slightly different process which is described on KernelTeam/KernelUpdates.

10.2. Landscape

The landscape-client source package may be uploaded according to the procedure documented in LandscapeUpdates. See the Technical Board resolution for details and rationale.

10.3. tzdata

The tzdata package is updated to reflect changes in timezones or daylight saving policies. The verification is done with the "zdump" utility. The first timezone that gets changed in the updated package is dumped with "zdump -v $region/$timezone_that_changed" (this needed to be greped for in /usr/share/zoneinfo/). This is compared to the same output after the updated package got installed. If those are different the verification is considered done.

Because tzdata's packaging has changed subtly from release to release, rather than just backporting the most recent release's source package, we just update the upstream tarball instead. You then need to edit debian/changelog to add bug closures, and make sure to use a version number consistent to the previous numbering scheme (e. g. 2012e-0ubuntu0.12.04).

11. Examples

As a reference, see bug #173082 for an idea of how the SRU process works for a main package, or bug #208666 for an SRU in universe.

12. Package Removals

While it is always preferable to fix a package, rather than drop it, there are rare cases when a universe package becomes actively detrimental in stable releases: If it is unmaintained in Ubuntu and has unfixed security issues or has been broken because of changing network protocols/APIs, it is better to stop offering it in Ubuntu altogether rather than continuing to encourage users to install it.

It is not technically possible to remove a package from a stable release, but this can be approximated by SRUing an essentially empty package with an appropriate explanation in NEWS and a corresponding critical debconf note.

When a package is removed in this way from a stable release, it may need similar removal from the devel release as well, depending on the justification for removal.

Such a package removal should have an SRU tracking bug with an appropriate explanation, and needs to get confirmed by the Technical Board. Once removed, the SRU bug should be added to the "Previous Removals" list below.

Previous Removals:

• tor (was reintroduced later on in #413657)

• bitcoin

• owncloud

13. Links

Bugs in different stages of the stable release process: http://people.canonical.com/~ubuntu-archive/pending-sru

• This page has an "Upload queue status" section which links to all stable review queues.

Phasing of Stable Release Updates: http://people.canonical.com/~ubuntu-archive/phased-updates.html

• This page displays the Phased-Update-Percentage of packages in the -proposed repository for releases and any regressions detected in that package.

14. Reviewing procedure and tools

If you are a member of the SRU reviewing team, you should check out the ubuntu-archive-tools scripts with

•  bzr branch lp:ubuntu-archive-tools

which greatly simplify the reviewing procedure. You should symlink sru-review and sru-accept somewhere to your ~/bin/ directory for easy access, or put the checkout into your $PATH.

The following review procedure is recommended:

• Open the unapproved queue for a particular release, e. g. https://launchpad.net/ubuntu/precise/+queue?queue_state=1 for precise. This shows the list of SRU uploads which have to be reviewed, commented on, and approved/accepted/rejected.

• For each package, generate the debdiff to the current version in the archive and open the corresponding bugs:

   sru-review -s saucy gnash

  This opens all the bugs which are mentioned in the .changes file in the browser, and will generate a debdiff between the current archive and the unapproved upload (unless the orig.tar.gz changes this will only download the two diff.gz, so it is reasonably fast).
• Review the bugs for complete description, justification, check that they have a stable release task, are conformant to SRU rules, etc, and comment accordingly.
• Scrutinize the debdiff for matching the changes in the bugs, not having unrelated changes, etc. If you have doubts, comment on the bug.

• If you are in the ubuntu-sru team:

  • Exit the tool you are using to review the debdiff

  • If the bugs and debdiff are okay, accept the package by pressing y at the ""Accept the package into -proposed?" prompt.

    This will tag the bug(s) with verification-needed, subscribe ubuntu-sru, and add a general "please test and give feedback"-like comment.

  • If the upload is broken or unsuitable for an SRU, reject it by pressing N at the ""Accept the package into -proposed?" prompt and pressing y at the "REJECT the package from -proposed?" prompt.

• If you are not in the ubuntu-sru team: Send a follow up comment to the bugs:

  • If all is okay: send an "ubuntu-sru approved and reviewed" comment and set the task to "In Progress"
  • If something is wrong: send the feedback to the bug and set the task to "Incomplete"

The pending SRUs should also be reviewed to see whether or not there are any to be released or removed from the archive. The process for dealing with these follows:

CategoryProcess