Stable Kernel Updates Maintenance

This page details how to maintain the stable patches releases provided by the Ubuntu Kernel Team. For more information, see Stable Kernel Updates

The maintenance process is comprised of these main parts:

  1. Adding patches to the queue
  2. Closing a set of patches to be released, and announcing a review round
  3. Doing and announcing a new release

Adding patches to the queue

After a new stable release, we should start picking new patches marked for stable on master Linus tree, and applying them to the -queue branch. This can be done at any time, and after 1 to 3 weeks, the patches applied to -queue must be pushed for review.

To apply patches to the queue, a script was created to make the task of grabbing patches that have "Cc: stable..." in its changelog easier, and may be optionally used. The script is stable/apply-stable-patches from git://

Before starting, make sure you have the latest kteam-tools.git checkout available. Then make sure you have the latest linux tree with stable branches checkout from git://, and go to the queue branch for the maintained version:

cd linux
git checkout linux-<version>.y-queue (this branch should track origin/linux‑<version>.y-queue)
git clean -dxf
git reset --hard origin/linux‑<version>.y-queue

Now you must identify the range of commits upstream you want to check that apply to stable. For example, lets say that on master Linus tree, the last patch added to the queue branch was 06383f10c49f507220594a455c6491ca6f8c94ab, and that for sure it is the last patch merged from Linus tree applied to the queue branch. So now we must apply any patch marked for stable after 06383f10c49f507220594a455c6491ca6f8c94ab. The apply-stable-patches script automatically fetches master from Linus tree, so to apply patches marked for stable to the current linux‑<version>.y-queue, run:

<path to kteam-tools.git checkout>/stable/apply-stable-patches \
   --range=06383f10c49f507220594a455c6491ca6f8c94ab..FETCH_HEAD \

Optionally, you can restrict the application to a commit previous to FETCH_HEAD (FETCH_HEAD points to master Linus tree head).

Note that the --sob option will add a 'Signed-off-by:' to all the patches. The full name and email address will be obtained from the git configuration and, and can be overwritten by the --name and --email options.

apply-stable-patches will check all commits in the range given, the range can be in any format which git understands, make sure you type a range that is in upstream Linus tree. While running, it will check for all commits marked for stable on that range, and try to apply it on the current linux‑<version>.y-queue branch. It will output all changes marked for stable and their application status. After the first run, 5 directories will be created in the current directory, where all patches with changes marked for stable are copied to some of them, depending on the application status:

  • Applied - will contain successfuly applied patches

  • Failed - will contain patches which failed to apply

  • Ignored - will contain patches not applied because the script checked its Cc: line, and it was specified for a different version than we are trying to apply

  • Discarded - nothing copied by the script to it, you will move here patches later you want to exclude on next run

  • Fixed - nothing copied here by the script, you will move fixed patches from Failed here

Now, you must fix or discard patches which failed to apply, and are in Failed directory. If the patch should be included or backported, you must fix the patch or get it fixed from someone (make sure you keep the same file name, and the original changelog with optional additions on top), and move it to the Fixed directory. If the patch should be discarded, move to the Discarded directory.

Note: When dealing with the failed patches, make sure you use the -k parameter to "git format-patch", otherwise the commit body (description) will get lost when re-running apply-stable-patches as this script will use "git am -k" to apply the patches.

After dealing with all Failed patches, reset the queue tree again (git reset --hard origin/linux‑<version>.y-queue), and run again apply-stable-patches with the same arguments under the linux‑<version>.y-queue checkout. The script will detect patches now in the Discarded and Fixed directories, and will ignore or apply the fixed patches. Note that the script uses "git format-patch -k" and "git am -k", so it detects the changes using the file name, that's why you can't change the file name when moving the patch to either Fixed or Discarded directories. Also ordering is ensured because the script apply patches in the order they were merged in the upstream range given. In case the fixed/discarded patches introduce problems on patches that applied before (and don't apply now, because it depended on a now removed patch etc.), it now will be copied to Failed directory, and you should deal with them, and again reset the current branch and run apply-stable-patches (you can reset the queue branch and run the script with same arguments as many times you want, until everything in Failed is handled).

With failed patches handled, review if patches in Ignored really aren't supposed to be applied. In case the script failed in some way and a change in Ignored should have been applied, move the patch file to Fixed, making sure it applies. Then reset the queue tree (git reset --hard origin/linux‑<version>.y-queue) and run apply-stable-patches again, the script will apply the change from the Fixed directory. Make sure there are no new failures because of the new introduced patch (patches in Failed).

With no failed patches, and any Ignored changes ok, we now will have all patches marked with Cc: stable on the given range applied to the current linux‑<version>.y-queue checkout. Review the branch, with everything ok you can push the changes (git push <remote> linux‑<version>.y-queue:linux‑<version>.y-queue). Note that the script automatically adds your sign-off, and a pointer to the upstream commit hash to the changelog of all applied patches.

After pushing, make sure to do any build testing/etc., verifying that everything is ok. Then extract the patches (eg. git format-patch -N origin/linux-3.5.y-queue..), and use <path to kteam-tools.git checkout>/stable/notify-stable-patch-queued *.patch, to send email notifications about the patches added to the queue.

Note that the apply-stable-patches script only adds patches marked with "Cc: stable..." on Linus tree. For other patches in Linus tree that are added to some upstream stable tree and should be added to this one, which were not marked with "Cc: stable", or any other specific case, like ones detailed in Contributing section at Stable Kernel Updates, patches must be applied/handled manually (and also don't forget to announce the addition with notify-stable-patch-queued).

Testing the stable kernels

It is desirable that the -queue (and the -review) branches are never broken, and that it is always possible to build and run them. For this to be achieved, this branch is regularly build and tested.

Daily builds to use with Ubuntu amd64, i386 and armhf can be obtained here For example, the latest daily build for the 3.5.y kernel -queue branch is available here These kernels are built using the Ubuntu kernel configuration for the corresponding release (e.g., Quantal kernel configuration for 3.5.y kernels, Raring kernel configuration for the 3.8.y kernels, etc).

Additionally, every time patches are queued into a kernel, they are also build tested with a make allmodconfig configuration.

Finally, there are regular test runs using the Ubuntu kernel team testing framework in order to reduce the risk introducing regressions.

Closing a set of patches to be released, and announcing a review round

Passing 1 to 3 weeks after previous release, we should prepare the release of the new set of patches on queue branch. Just apply them to the review branch in any way you want, and push. Then use the script stable/notify-stable-review from git:// to send an announcement email to the relevant mailing lists:

  • Make sure you are on the linux‑<version>.y-review git branch you just pushed, and that your git config has any sendmail configuration if necessary already added (like sendemail.config and sendemail.identity)

  • Run  <path to kteam-tools.git checkout>/stable/notify-stable-review --dry-run --previous_release=<previous HEAD> --new_version=<version string> if you want to review the email that is going to be sent

  • Run  <path to kteam-tools.git checkout>/stable/notify-stable-review --previous_release=<previous HEAD> --new_version=<version string> to send the announcement email.

Note: the script automatically extracts and sends the patches for review from the current linux‑<version>.y-review checked out. Make sure the --previous_release=<previous HEAD> parameter given is correct (previous HEAD is any git reference, eg. origin/linux-3.5.y, or the previous tag eg. if we are starting the review of 3.5.7-ext.2, the paramenter given can be v3.5.7-ext.1). --new_version is the version string that will be displayed on emails, the new version for review being announced.

If the script should happen to fail part-way through a run (say, if one of the patches gets parsed badly by git-send-email) you can use the options "--edit_patch_dir --sent_cover_letter=<Message-Id>" to recover. First, fix and commit the problem patch (rebase and push the fixed -review branch). Then run the notify-stable-review again with the same command line options, plus those two. Provide --sent_cover_letter=<Message-Id> with the Message-Id: from the cover letter message that already got sent out (omit the <angle> brackets). The --edit_patch_dir option will start a sub-shell in a tmpdir filled with all of the patches in the series; remove the files corresponding to the patches that already got sent out, then exit the sub-shell; only the remaining patches will then be sent, as replies to the original cover letter. Using --dry-run along with these recovery options works too, and is highly recommended.

After you send the announcement, pay attention to any review feedback, and fix the problems that are reported. Review time should be 3 days, then you can go to next step, doing and announcing the new release.

Doing and announcing a new release

With review done, and everything fixed, push the patches in review to the main linux‑<version>.y branch, close the series and tag the new version:

git checkout linux‑<version>.y
git clean -dxf
git reset --hard linux‑<version>.y
git merge linux‑<version>.y-review
<edit Makefile, increment SUBLEVEL or EXTRAVERSION, what is applicable to as the new version>
git add Makefile
git commit -s -m "Linux <version>"
git tag -s -m v<version> v<version>
git push <remote> linux‑<version>.y:linux‑<version>.y v<version>

Then sync the -queue and -review branches, for example:

git push <remote> linux‑<version>.y:linux‑<version>.y-queue
git push <remote> linux‑<version>.y:linux‑<version>.y-review

Finally, send an announcement email to the relevant mailing lists, about the new version. To do it, use the script stable/notify-stable-release from git://

  • Make sure you are on the linux‑<version>.y git branch you just pushed, and that your git config has any sendmail configuration if necessary already added (like sendemail.config and sendemail.identity)

  • Run  <path to kteam-tools.git checkout>/stable/notify-stable-release --dry-run --previous_release=<previous HEAD> --new_version=<version string> if you want to review the email that is going to be sent

  • Run  <path to kteam-tools.git checkout>/stable/notify-stable-release --previous_release=<previous HEAD> --new_version=<version string> to send the announcement email.

Kernel/Dev/ExtendedStable/Maintenance (last edited 2013-08-01 17:18:12 by kamalmostafa)