Ubuntu Wiki

Overview

The work done as part of the stable maintenance team basically fall into two main categories:

1. Stable release updates (SRUs)
2. Security updates (CVEs)

Most of the complexity today results of the fact that both tasks are not only done for the main kernel package of all supported releases, but also for each of the topic branches within each release. Also every release can have slight differences in the way things are done or which packages depend on ABI (application binary interface) changes of the kernel. (see the KernelChecklist for some more details on that).

Stable Release Updates

SRUs are generally allowed for everything that is a bug. However, in order to prevent regressions as much as possible, several restrictions apply.

The general SRU process is described here. For the kernel, because it is a critical part of the system we decided to make a few modifications as bugs in the kernel often cannot be worked around and the expectations are rather to have problems fixed in the current release than to wait for the next one (within certain limits). On the other hand extreme care has to be used when making any changes because the kernel is a critical part of the system.

The SRU Rules

[should be a reference to the public section, need to move https://wiki.ubuntu.com/KernelTeam/KernelUpdates to a sensible place and limit to generic policy there]

Additionally to the generic requirements acceptable bugs fall into one of the following criteria:

1. It fixes a critical issue (data-loss, OOPs, crashes) or is security related (security related issues might be covered by security releases which are special in handling and publication).

2. Simple, obvious and short fixes or hardware enablement patches as long as the next release has not reached beta status. If there is a related upstream stable tree open (see below) this class of patches is required to come through the upstream process. Patches sent upstream for that reason must include their BugLink reference.

3. For 3 months after a non-LTS release or the lifetime of a LTS release we will be pulling upstream stable updates from the corresponding series. There will be one tracking bug report for each stable update but additional references to existing bugs will be added to the contained patches (on best can do base). Uploads containing a upstream stable patchset increase the retention period to two weeks, while it is one week otherwise.
4. Fixes to drivers which are not upstream are accepted directly if they fall into the first two categories.

5. $DEITY intervention. Might happen, but very very rarely and will not be explainable.

The SRU Rules - explained

Because we release every six months, there is a natural time limit on how long we can address bugs in a released kernel. So we now say to fix bugs for 4 months after release. Though this means that after three months the bug work should move on to concentrate on bugs still visible in the development release and fixes to the released kernel are only backports as a fix done in the development kernel.

Upstream stable updates fits into this quite well. Usually a upstream stable tree is only maintained for about 3 months. After that there is a new release by Linus and the tree is retired in favor of the new version. Also the criteria for accepting patches into the upstream stable tree are quite similar to our own. And when things are added there it is seen and reviewed by a larger audience from the community. So at some point the decision was made to try to get all bug fixes that we want to make, accepted and included upstream and to move forward and apply all patches coming from upstream wholesale. While this creates a lot of work to submit things, argue and review as much as possible, the advantage is that this increase the visibility of us to "give back" and also makes sure, that there are not patches lost when moving on to the next release.

Surely there will be a few exceptions and special handling. For critical bugs we likely want to add them to the kernel without waiting for the stable release. And sometimes we carry special modifications anyway. Like more recent subsystems or actually drivers which are not upstream (yet). In those cases it still should be possible to fix a bug.

LTS release have some additional modifications to the SRU rules. First, upstream stable updates are allowed for the whole lifetime of the release. This does not necessarily change much. For Hardy, upstream stopped doing stable updates after 3 months because that is what they usually do. Lucid is the first release that will receive real long term upstream support because this is an upstream long term supported release for them as well. Second, we fix critical bugs during the lifetime of the release as well (hoping there are not too many detected after the initial 3 months). And finally, to allow support for newer hardware during support time, we allow for a limited amount of enablement patches. This is intended mainly for adding IDs or quirks, for which the regression potential is very low.

Workflow for SRU Patches

1. Have a Launchpad bug report
   Usually there is but sometimes people just send patches. Always create (or let create) a bug report. We need to reference that in the upload.

2. Get the solution tested
   This will not prevent regressions for others but at least makes sure it works.

3. If the patch is not upstream, send it upstream. If applicable send it as stable fix (KernelTeam/StableHandbook/UpstreamStableSubmission). If it is upstream, could be stable material, but has not marked as stable patch, get back to the subsystem maintainer and try to get it submitted (or offer to submit it).

4. Add a SRU justification to the bug report. The mantra is IFT: Impact, Fix, Testcase.
   • Impact: What are the symptoms/severity of the problem.
   • Fix: Where does it come from? Why is it not upstream if it is not? Has it been submitted for stable? How big seem the chances for regressions?
   • Testcase: How can the issue be reproduced? Quite often this requires certain hardware. Best place for the justification is in the bug description. That makes it immediately visible when looking at the bug.

5. Send email with the justification and the patches
   For larger patches the better form is a git pull request. But it is still helpful to have some patch included with that. The reason is that patchwork (http://patchwork.ozlabs.org) will only pick up mails that seem to contain patches.

   Include BugLink and your signed-off-by (SOB) in patches you send around. Also reference where this patch comes from ("cherry-picked from commit # upstream" or "backported from commit # upstream").

6. Wait for two ACKs
7. When in charge of the repository and after two ACKs have arrived, take the patch, add the Acked-by lines and your SOB. And commit the patch(es). Then push.
8. Rinse and repeat.

Workflow for Pulling in Upstream Stable Releases

KernelTeam/StableHandbook/UpstreamStableReview

<branch>-next as a holding place

Each kernel git repository has a -next branch, usually master-next, which is used as a holding place for patches that have not been applied to <branch> while <branch> is undergoing testing within the stable kernel cycle. In order for patches to be applied to <branch>-next they must have the requisite number of ACKs (stable updates excluded). Any developer in the kernel_cdev group can push to this branch. Periodically <branch>-next is rebased onto <branch> by a stable updates team member, typically at the start of the stable kernel cycle. Pre-proposed builds are also built from master-next in order to get maximum test time. Security patches are also applied to master-next, particularly those CVE patches that have been promoted to -security so that the next upload to -proposed does not regress with respect to CVE support.

Security Updates

Security flaws are tracked by the security team in a bazar branch. We take a branch of that, which is owned by the canonical-kernel-team and use that to look for individual tasks and also to coordinate assignments. It is also a back channel for the security team.

• bzr launchpad-login
• bzr pull lp:~canonical-kernel-team/ubuntu-cve-tracker/kernel-team
• update status fields in active/*
• bzr commit -m"Updated CVE-2010-*"
• bzr push lp:~canonical-kernel-team/ubuntu-cve-tracker/kernel-team

Previously the security team would collect a certain number of CVE bugs and then address the kernel team for triage and patch application. To avoid unnecessary ABI bumping, this would be targeted to happen when no ABI bumping proposed kernel was pending. However this could still result in a time consuming process, depending on the complexity of patches involved.

See SecurityReleaseVersioning for an explanation on the branching and version numbering of security uploads.

So now, the new process addresses this by allowing concurrent work on the separate CVEs by different members of the team in advance. Whenever a release should be done, it will just pull together the work that is ready and merge it into release trees.

The process will also make sure the trees of various stages are available centrally without being public (needed if embargoed issues are handled). A single person will be in charge of the merging and preparation for release.

A detailed explanation is found in the KernelTeam/StableHandbook/SecurityProcess section.

Best Practices

• The first upload to -proposed, -security, or -updates after an official release should always bump the ABI. This ensures that a user who installs from the CD has a known good fallback kernel in the event the first kernel update introduces any regressions.