Differences between revisions 11 and 12
Revision 11 as of 2010-07-28 15:18:05
Size: 14991
Editor: sconklin
Revision 12 as of 2010-07-28 15:33:29
Size: 10906
Editor: sconklin
Deletions are marked like this. Additions are marked like this.
Line 15: Line 15:
=== Upstream stable fixes ===

As soon as a new upstream kernel is released by Linus, a related 2.6.x.y stable maintenance tree is established to deliver updates to the released kernel. The associated stable tree is maintained until the next upstream kernel is released, unless the kernel has been designated for long term support by the kernel community. These trees are maintained by [[|Greg Kroah-Hartman]]. The kernels currently designated for long-term support are 2.6.27 and 2.6.32.

The git repository containing branches for each upstream stable kernel is here:
Line 26: Line 17:
=== Getting Fixes into upstream stable ===

Any fix to a released upstream kernel which is of general interest should be submitted for possible inclusion in the upstream stable tree.

There is a documentation file on the upstream stable process which can be found in the Linux tree under ''Documentation/stable_kernel_rules.txt''. The rules for acceptance of patches are similar to those for SRU acceptance in the Ubuntu kernel (below). Though in practice, the rules are sometimes relaxed.

==== New patches ====

When trying to get a fix into upstream stable, determine first whether the patch has already been submitted to upstream stable for consideration. When an author or subsystem maintainer determines that a patch submitted to Linus for the mainline kernel should also be considered for stable updates, they will add the following line to the signed-off-by section of the patch, will cause it to be pulled into the currently maintained stable trees as soon as it is pulled into Linus tree. If a patch has been CC'd to stable, then there is no need to resubmit it. If you are proposing inclusion of such a patch for Ubuntu, please state that it has been submitted for stable in the bug report and email to the ubuntu-kernel-team.

Signed-off-by: ...

==== Fixes in Linus's mainline tree but not in upstream stable ====

If the patch is already in Linus's tree, but has not been sent to stable, it can be sent to the stable mail address. When sending patches, you should CC the subsystem maintainer (and all those that have been in the signed-off-by section). The mail must contain a quick explanation of why this meets the stable kernel rules. Generally the reason a patch qualifies is that it has been proven to fix an observed (not theoretical) bug. Depending on the patch, it might be a good idea to contact the subsystem maintainer on the subject and get his or her agreement in advance.

==== Different fix than upstream ====

There are cases where a fix will be different from the patch submitted to the upstream tree. This can be because the patch no longer applies cleanly to an older kernel and requires some changes, or because the patch is a fix that is not required due to redesign of a subsystem. In these cases the backported or specially crafted patch is sent to directly, as described above. The subsystem maintainer MUST be copied, and it is strongly suggested that you contact the subsystem maintainer first, and perhaps ask them to propose the fix for inclusion in upstream stable. Your mail must clearly explain why your fix is done the way it is, or the patch could be ignored.

==== Hinting pre-requisite patches ====

As a new addition to the known "Cc:" line, it is possible to give hints about any patches that are required for the stable patch to work correctly. The following example, taken from the stable_kernel_rules.txt file, shows the signed-off-by section in such a case:

Cc: <> # .32.x: a1f84a3: sched: Check for idle
Cc: <> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <> # .32.x: fd21073: sched: Fix affinity logic
Cc: <> # .32.x
Signed-off-by: Ingo Molnar <>

Patches will be picked in the order they appear, so a1f84a3 will be the first and fd21073 the last (before the patch this is contained is applied).
Please see UpstreamStableKernel for more information about that process.

This document is intended to document the processes involved in the maintenance of stable (after release) kernel trees. Those should be independent of a specific release. If not, this should be noted on the specific description.

Post-release updates

The vast majority of fixes to released Ubuntu kernels come through the upstream stable fixes process. A minority of fixes originate from Ubuntu developers. Any fixes originating from Ubuntu developers should be submitted upstream, so that the entire Linux community will benefit.

Updates to the upstream stable trees are automatically considered for inclusion in Ubuntu Stable Release Updates (SRUs).

Please see UpstreamStableKernel for more information about that process.


Stable release updates will be done after the initial release. To prevent regressions there are a more formal process involved in getting new patches included. The process is described in general at StableReleaseUpdates with a kernel specific part described KernelUpdates.

  • Every SRU needs a bug. Patches are sent to kernel mailing list and have to get at least two ACKs from other senior kernel developers (those in the kernel_cdev group on zinc).

  • All patches should be signed-off-by the person who did the patch. The developer who is committing the final patch, also needs to add his SOB.
  • Patches from upstream should not change the submitter's reference. They should be cherry-picked or exported and then re-imported (git format-patch and git-am. Personally I do a

    git format-patch -s <sha> -1
    then add bug and/or CVE references and then import the resulting patch (the -s automatically adds a signed-of-by line).
  • Changes that require an kABI bump should be marked by adding "BumpABI: yes" to the git message. This is a manual process but makes life simpler of someone asks why the ABI has been bumped.
  • SRU justification is added to bug report and nomination for releases are used to track progress for each release (if appropriate).
  • ubuntu-sru has to get subscribed
  • The milestone is set for the next official update release only for bugs that are important (iow when you want to be chased by a release manager if that patch fails to get included).

CVEs/security updates

Current contact: Kees Cook

CVE updates usually are done in batches (except for very critical problems). There is a list of CVEs and all currently maintained releases have to be checked for possible vulnerabilities.

The patchsets are prepared by one member of the stable maintenance team but not pushed if any of the patches still is embargoed.

  • test builds must be done at least for i386 and amd64
  • test packages and upload sources are provided on an internal server (e.g. chinstrap).
  • boot tests should be done for i386 (and amd64)
  • Kees has a little test script in bazar which is good for testing:

    bzr branch lp:~ubuntu-bugcontrol/qa-regression-testing/master qa-regression-testing
    cd qa-regression-testing/scripts
    sudo apt-get install build-essential python-unit sudo libcap2-bin lsb-release
    ./ -v

Work flow for security updates with unreleased or pending -proposed updates:

  • create a branch based on the last official (-updates/-security) release
  • add fixes and do the release from this branch
  • after the security release is final and if there was a release in proposed, create a new branch which is based on the last proposed release, merge back the security branch and prepare everything for a rebuild (fix up ABI files and the changelog).
  • Finally check out the master branch, merge back either the security or proposed branch and fix up the ABI and changelog.

This makes sure the version of the security release can be checked out and based upon.

Note: Had to use "git push origin master" to get merges visible.


A general rule is that the security release has to be a higher version/release number than anything else. This causes some headaches if there is a proposed version not yet moved into updates, especially if this is an ABI bumper.

  1. If there is no kernel package in proposed, the security release just gets the next release number. Should the security release require an ABI bump, this is also done like normally.
  2. If there is a kernel in proposed, the security release gets the next number that would follow it.
  3. Should this proposed kernel be an ABI bumper, then the security release has to match that ABI number and the rebuild of proposed needs to bump the ABI again.
  4. Should the proposed kernel be an ABI bump and the security release is also an ABI bump, the security release must bump the ABI and upload number of the kernel currently in proposed.

The following examples try to illustrate the version handling:

Updates:  1.1 --+-------------------------+--
Proposed:        \-- 1.2 -------+-- 1.4 --/
Security:         \-------- 1.3 /

Updates:  1.1 --+--------------------------+--
Proposed:       \-- 2.2 ---------+-- 3.4 --/
Security:        \-------- 2.3 --/

Updates:  1.1 --+----------------------------+--
Proposed:        \-- 2.2 ----------+-- 4.4 --/
Security:         \--------- 3.3 --/

In any case where there was a proposed kernel pending and it has to be rebuild with the security patches included, the changelog will be modified to include all changes that have been recorded by proposed uploads which have not been moved to updates, as the new release and the old ones get dropped. Here an example. Given there have been two uploads to proposed accumulated but not moved to updates. The changlog would look like this:

linux (2.6.24-1.3) hardy-proposed ...
  - item 3
linux (2.6.24-1.2) hardy-proposed ...
  - item 2
  - item 1
linux (2.6.24-1.1) hardy ...
  - in updates

Then a security release comes and takes precedence. After that has been released we prepare a new upload to proposed and the changelog looks like that:

linux (2.6.24-1.5) hardy-proposed ...
  - item 3
  - item 2
  - item 1
linux (2.6.24-1.4) hardy-security ...
  - security update
linux (2.6.24-1.1) hardy ...
  - in updates

Updates to custom binaries (virtual/openvz/...)

Custom binary builds are usually community maintained and not as critical as the main kernel flavours. Nevertheless updates should follow a certain pattern to be clearly documented.

  • Get triggered by a community member (usually kernel-team mailing list)
  • Open a LP bug for SRU documentation
  • Create a branch and pull/cerry-pick
  • Do a test build
  • Publish the SRU request on the kernel-team list. References to patches as gitweb links since patches might be big.

Test builds

Either in a PPA or on the porter machines. HPPA is a problem due to hardware access.


Pre-release tasks

A new upload is prepared by editing/creating the changelog (KernelMaintenance)

  • Use chroots to create the uploads (because scripts depend on certain environments)
  • The insertchanges target does not always work properly, especially on older releases. Always make sure to check the generated file.

  • Are there duplicate/unwanted entries?
  • Has every entry a bug reference?
    • Ubuntu patches (UBUNTU keyword prefixed) automatically get this if they contained a Bug keyword in the description. Some of the build environments have been updated to do the same for upstream patches with a Bug entry.

    • Patches from upstream did not (but do with updated scripts and if they include a Bug keyword.

    • CVEs should have the CVE-xxxx-xxxx number in their description
      •       UBUNTU: Fix stupid bug
  • The changelog keeps order of uploads, git log the order of check-ins but this cannot always be honoured. The make process cannot handle complicated forth and back of ABI releases. But in general the sequence of the changelog reflects the upload history

On updates the command for generating the source package should include the -v option (and have the orig.tar.gz, to do only diff uploads). If the the orig.tar.gz file is placed into the directory above the kernel tree, this will be done automagically.

dpkg-buildpackage -S -rfakeroot -I.git -I.gitignore -i'\.git.*' \

A note to <version>: this should be the last relevant version uploaded. So for uploads to -proposed this should be the last version uploaded to updates. Updates to security should probably be done the same way, but since -security is moved to -updates rather quickly, both will be the same version anyway.

The actual release

The release is done by uploading the source packages to special servers. If the kernel ABI has changed, LUM, LBM and LRM must be uploaded with updated version/ABI numbers as well. Also the linux-meta package must be updated and uploaded (after the other packages are done).

  • The upload target differs, depending on the type of upload:
    • -security uploads go to (jackass) while

    • -proposed uploads go to

  • The packages have to be signed by someone with the upload rights and the uploaded packages will be invisible until they are ACCEPTED

    • For -security uploads the contacts are kees or jdstrand. For those the status is even invisible after acceptance.

    • For -proposed uploads the packages have to be accepted by either slangasek or pitty (or cjwatson(?))

  • If the kABI changed note the reasoning on the stable kernel ABI wiki page. This must also be announced by mails to

  • Somehow also keep track of the changes in supported hardware to gather this for releases.

Post-release tasks

As described in KernelMaintenance. Thoughts on that:

  • The ABI files have to be fetched when starting a new release. The buildscript build-getabis can be used to fetch the ABI files from the builders. So one has not to wait until the upload is complete.

KernelTeam/StableKernelMaintenance (last edited 2016-07-07 19:32:24 by kamalmostafa)