StableKernelMaintenance

Differences between revisions 1 and 15 (spanning 14 versions)
Revision 1 as of 2008-11-04 11:28:05
Size: 6943
Editor: p5B2E44A4
Comment:
Revision 15 as of 2011-03-03 02:16:39
Size: 8365
Editor: pool-98-108-155-157
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
## page was renamed from StableKernelMaintenance
Line 12: Line 13:
The vast majority of fixes to released Ubuntu kernels come through the upstream stable fixes process. A minority of fixes originate from Ubuntu developers. Any fixes originating from Ubuntu developers should be submitted upstream, so that the entire Linux community will benefit.

Updates to the upstream stable trees are automatically considered for inclusion in Ubuntu Stable Release Updates (SRUs).

Please see [[KernelTeam/UpstreamStableKernel|UpstreamStableKernel]] for more information about that process.
Line 17: Line 24:
StableReleaseUpdates with a kernel specific part described KernelUpdates (''Note the proposal below''). StableReleaseUpdates with a kernel specific part described [[KernelTeam/KernelUpdates|KernelUpdates]].
Line 19: Line 26:
  * Every SRU needs a bug. Patches are sent to kernel mailing list and have to get at least one ''ACK'' (maybe 2 with the current team size).   * Every SRU needs a bug. Patches are sent to kernel mailing list and have to get at least two ''ACK''s from other senior kernel developers (those in the kernel_cdev group on zinc).
Line 21: Line 28:
  * SRU justification is added to bug and nomination for releases are used to track progress for each release (if appropriate).   * All patches should be signed-off-by the person who did the patch. The developer who is committing the final patch, also needs to add his SOB.

  * Patches from upstream should not change the submitter's reference. They should be cherry-picked or exported and then re-imported (''git format-patch'' and ''git-am''. Personally I do a {{{
git format-patch -s <sha> -1
}}} then add bug and/or CVE references and then import the resulting patch (the -s automatically adds a signed-of-by line).

  * Changes that require an kABI bump should be marked by adding "BumpABI: yes" to the git message. This is a manual process but makes life simpler of someone asks why the ABI has been bumped.

  * SRU justification is added to bug report and nomination for releases are used to track progress for each release (if appropriate).
Line 25: Line 40:
  * ['''TBD'''] milestone is set for ??? (patches that are important, or just generally if we think it should get into next release).

'''Proposal:''' Always add fixes from -stable tree.

Since the process for stable kernel updates ([[http://lwn.net/Articles/126915/|LWN article]])
is very similar to our SRU process and there have been quite a number of bugs
that have been solved by finally finding out that there has been a fix in
the stable tree, there has been a proposal to add those automatically to our
stable tree (discussion was on kernel team mailing list). The updated process
could look like this:

  * Stable release updates from the kernel.org stable kernel that matches the kernel version will qualify as SRU updates. Possible import process could be:

    * one LP bug for each release
    * one LP bug for each patch of each release

  * Those updates also should be ''ACK''ed by at least one (two) members of the kernel team.

  * Updates resulting from stable tree imports should be kept at least 2 weeks in -proposed before moving to -updates. Any change or regression in the stable tree resets the counter.
  * The milestone is set for the next official update release only for bugs that are important (iow when you want to be chased by a release manager if that patch fails to get included).
Line 48: Line 44:
Current contact: ''Kees Cook'' See: [[https://wiki.ubuntu.com/Kernel/Dev/FixingCVEs#preview| HowTo: Fixing CVEs ]]
Line 50: Line 46:
CVE updates usually are done in batches (except for very critical problems).
There is a list of CVEs and all currently maintained releases have to be
checked for possible vulnerabilities.
=== Versioning ===
Line 54: Line 48:
The patchsets are prepared by one member of the stable maintenance team but
not pushed if any of the patches still is embargoed.
A general rule is that the security release has to be a higher version/release number than anything else. This causes some headaches if there is a proposed version not yet moved into updates, especially if this is an ABI bumper.
Line 57: Line 50:
  * test builds must be done at least for i386 and amd64  1. If there is no kernel package in proposed, the security release just gets the next release number. Should the security release require an ABI bump, this is also done like normally.
 1. If there is a kernel in proposed, the security release gets the next number that would follow it.
 1. Should this proposed kernel be an ABI bumper, then the security release has to match that ABI number and the rebuild of proposed needs to bump the ABI again.
 1. Should the proposed kernel be an ABI bump and the security release is also an ABI bump, the security release must bump the ABI and upload number of the kernel currently in proposed.
Line 59: Line 55:
  * test packages and upload sources are provided on an internal server (e.g. chinstrap). The following examples try to illustrate the version handling:
Line 61: Line 57:
  * boot tests should be done for i386 (and amd64) {{{
Updates: 1.1 --+-------------------------+--
Proposed: \-- 1.2 -------+-- 1.4 --/
Security: \-------- 1.3 /
Line 63: Line 62:
  * ['''TBD'''] verification of the vulnerability (if possible)
    ''this would be good but requires automated tests''
Updates: 1.1 --+--------------------------+--
Proposed: \-- 2.2 ---------+-- 3.4 --/
Security: \-------- 2.3 --/
Line 66: Line 66:
Proposed work flow for security updates with unreleased or pending -proposed
updates:
Updates: 1.1 --+----------------------------+--
Proposed: \-- 2.2 ----------+-- 4.4 --/
Security: \--------- 3.3 --/
}}}
Line 69: Line 71:
  * create a branch based on the last official (-updates/-security) release In any case where there was a proposed kernel pending and it has to be rebuild with the security patches included, the changelog will be modified to include all changes that have been recorded by proposed uploads which have not been moved to updates, as the new release and the old ones get dropped. Here an example. Given there have been two uploads to proposed accumulated but not moved to updates. The changlog would look like this:
Line 71: Line 73:
  * add fixes and do the release from this branch {{{
linux (2.6.24-1.3) hardy-proposed ...
  - item 3
linux (2.6.24-1.2) hardy-proposed ...
  - item 2
  - item 1
linux (2.6.24-1.1) hardy ...
  - in updates
}}}
Line 73: Line 83:
  * after release (and after gathering the ABI files) merge back to main tree. Then a security release comes and takes precedence. After that has been released we prepare a new upload to proposed and the changelog looks like that:
Line 75: Line 85:
This makes sure the version of the security release can be checked out
and based upon.

'''Note:''' Had to use "git push origin master" to get merges visible.
{{{
linux (2.6.24-1.5) hardy-proposed ...
  - item 3
  - item 2
  - item 1
linux (2.6.24-1.4) hardy-security ...
  - security update
linux (2.6.24-1.1) hardy ...
  - in updates
}}}
Line 111: Line 127:
  * Are there duplicate/unwanted entries   * Are there duplicate/unwanted entries?
Line 113: Line 129:
  * Has every entry a bug reference   * Has every entry a bug reference?
Line 115: Line 131:
    * Ubuntu patches (UBUNTU keyword prefixed) automatically this if they contained a ''Bug'' keyword in the description.     * Ubuntu patches (UBUNTU keyword prefixed) automatically get this if they contained a ''Bug'' keyword in the description. Some of the build environments have been updated to do the same for upstream patches with a ''Bug'' entry.
Line 120: Line 136:
      {{{
      UBUNTU: Fix stupid bug

      BugLink: http://bugs.launchpad.net/bugs/<bugnumber>
      CVE-2008-0815
      }}}
Line 121: Line 143:
  * The changelog keeps order of uploads, git log the order of check ins but this cannot always be honoured. The make process cannot handle complicated forth and back of ABI releases.   * The changelog keeps order of uploads, git log the order of check-ins but this cannot always be honoured. The make process cannot handle complicated forth and back of ABI releases. But in general the sequence of the changelog reflects the upload [[https://edge.launchpad.net/ubuntu/+source/linux/+publishinghistory|history]]
Line 124: Line 146:
the -v option (and have the orig.tar.gz, to do only diff uploads) the -v option (and have the orig.tar.gz, to do only diff uploads). If the the orig.tar.gz file is placed into the directory above the kernel tree, this will be done automagically.
Line 133: Line 155:
to updates. Updates to security probably the same but most of the time
those are
both the same anyway.
to updates. Updates to security should probably be done the same way, but since -security is moved to -updates rather quickly, both will be the same version anyway.
Line 138: Line 159:
  * The upload target differs, depending on the type of upload:

    * -security uploads go to ''security.ubuntu.com'' (jackass) while

    * -proposed uploads go to ''upload.ubuntu.com''

  * The packages have to be signed by someone with the upload rights and the uploaded packages will be invisible until they are ''ACCEPTED''

    * For -security uploads the contacts are ''kees'' or ''jdstrand''. For those the status is even invisible after acceptance.

    * For -proposed uploads the packages have to be accepted by either ''slangasek'' or ''pitty'' (or ''cjwatson''(?))

  * If the kABI changed note the reasoning on the [[https://wiki.ubuntu.com/KernelTeam/Stable_kABI|stable kernel ABI]] wiki page.

    * Propose tag for ABI changing checkings (BumpABI). This is manual but it allows a quick overview which changes caused ABI bumps.

  * Somehow also keep track of the changes in supported hardware to gather this for releases.
The release is done by uploading the source packages to the kernel team ppa. Once it has been built, it get's pocket copied from the ppa to the -proposed pocket.
If the kernel ABI has changed, LUM, LBM and LRM must be uploaded with updated version/ABI numbers as well. Also the linux-meta package must be updated and uploaded (after the other packages are done).
Line 158: Line 164:
As described in KernelMaintenance. Thoughts on that: As described in KernelTeam/KernelMaintenance. Thoughts on that:
Line 160: Line 166:
  * The ABI files have to be fetched. Would it make sense to get them, then change the release tag to that state? Better would be to get those from test builds. Currently the only problem is HPPA (no host).

  * The creation of the new release uses ''whoami'' and stuff. Might be better to take DEBNAME and DEBEMAIL...
  * The ABI files have to be fetched when starting a new release. The maint-startnewrelease script can be used to fetch the ABI files from the builders.

This document is intended to document the processes involved in the maintenance of stable (after release) kernel trees. Those should be independent of a specific release. If not, this should be noted on the specific description.

Post-release updates

The vast majority of fixes to released Ubuntu kernels come through the upstream stable fixes process. A minority of fixes originate from Ubuntu developers. Any fixes originating from Ubuntu developers should be submitted upstream, so that the entire Linux community will benefit.

Updates to the upstream stable trees are automatically considered for inclusion in Ubuntu Stable Release Updates (SRUs).

Please see UpstreamStableKernel for more information about that process.

SRUs/bugs

Stable release updates will be done after the initial release. To prevent regressions there are a more formal process involved in getting new patches included. The process is described in general at StableReleaseUpdates with a kernel specific part described KernelUpdates.

  • Every SRU needs a bug. Patches are sent to kernel mailing list and have to get at least two ACKs from other senior kernel developers (those in the kernel_cdev group on zinc).

  • All patches should be signed-off-by the person who did the patch. The developer who is committing the final patch, also needs to add his SOB.
  • Patches from upstream should not change the submitter's reference. They should be cherry-picked or exported and then re-imported (git format-patch and git-am. Personally I do a

    git format-patch -s <sha> -1
    then add bug and/or CVE references and then import the resulting patch (the -s automatically adds a signed-of-by line).
  • Changes that require an kABI bump should be marked by adding "BumpABI: yes" to the git message. This is a manual process but makes life simpler of someone asks why the ABI has been bumped.
  • SRU justification is added to bug report and nomination for releases are used to track progress for each release (if appropriate).
  • ubuntu-sru has to get subscribed
  • The milestone is set for the next official update release only for bugs that are important (iow when you want to be chased by a release manager if that patch fails to get included).

CVEs/security updates

See: HowTo: Fixing CVEs

Versioning

A general rule is that the security release has to be a higher version/release number than anything else. This causes some headaches if there is a proposed version not yet moved into updates, especially if this is an ABI bumper.

  1. If there is no kernel package in proposed, the security release just gets the next release number. Should the security release require an ABI bump, this is also done like normally.
  2. If there is a kernel in proposed, the security release gets the next number that would follow it.
  3. Should this proposed kernel be an ABI bumper, then the security release has to match that ABI number and the rebuild of proposed needs to bump the ABI again.
  4. Should the proposed kernel be an ABI bump and the security release is also an ABI bump, the security release must bump the ABI and upload number of the kernel currently in proposed.

The following examples try to illustrate the version handling:

Updates:  1.1 --+-------------------------+--
Proposed:        \-- 1.2 -------+-- 1.4 --/
Security:         \-------- 1.3 /

Updates:  1.1 --+--------------------------+--
Proposed:       \-- 2.2 ---------+-- 3.4 --/
Security:        \-------- 2.3 --/

Updates:  1.1 --+----------------------------+--
Proposed:        \-- 2.2 ----------+-- 4.4 --/
Security:         \--------- 3.3 --/

In any case where there was a proposed kernel pending and it has to be rebuild with the security patches included, the changelog will be modified to include all changes that have been recorded by proposed uploads which have not been moved to updates, as the new release and the old ones get dropped. Here an example. Given there have been two uploads to proposed accumulated but not moved to updates. The changlog would look like this:

linux (2.6.24-1.3) hardy-proposed ...
  - item 3
linux (2.6.24-1.2) hardy-proposed ...
  - item 2
  - item 1
linux (2.6.24-1.1) hardy ...
  - in updates

Then a security release comes and takes precedence. After that has been released we prepare a new upload to proposed and the changelog looks like that:

linux (2.6.24-1.5) hardy-proposed ...
  - item 3
  - item 2
  - item 1
linux (2.6.24-1.4) hardy-security ...
  - security update
linux (2.6.24-1.1) hardy ...
  - in updates

Updates to custom binaries (virtual/openvz/...)

Custom binary builds are usually community maintained and not as critical as the main kernel flavours. Nevertheless updates should follow a certain pattern to be clearly documented.

  • Get triggered by a community member (usually kernel-team mailing list)
  • Open a LP bug for SRU documentation
  • Create a branch and pull/cerry-pick
  • Do a test build
  • Publish the SRU request on the kernel-team list. References to patches as gitweb links since patches might be big.

Test builds

Either in a PPA or on the porter machines. HPPA is a problem due to hardware access.

Release

Pre-release tasks

A new upload is prepared by editing/creating the changelog (KernelMaintenance)

  • Use chroots to create the uploads (because scripts depend on certain environments)
  • The insertchanges target does not always work properly, especially on older releases. Always make sure to check the generated file.

  • Are there duplicate/unwanted entries?
  • Has every entry a bug reference?
    • Ubuntu patches (UBUNTU keyword prefixed) automatically get this if they contained a Bug keyword in the description. Some of the build environments have been updated to do the same for upstream patches with a Bug entry.

    • Patches from upstream did not (but do with updated scripts and if they include a Bug keyword.

    • CVEs should have the CVE-xxxx-xxxx number in their description
      •       UBUNTU: Fix stupid bug
        
              BugLink: http://bugs.launchpad.net/bugs/<bugnumber>
              CVE-2008-0815
  • The changelog keeps order of uploads, git log the order of check-ins but this cannot always be honoured. The make process cannot handle complicated forth and back of ABI releases. But in general the sequence of the changelog reflects the upload history

On updates the command for generating the source package should include the -v option (and have the orig.tar.gz, to do only diff uploads). If the the orig.tar.gz file is placed into the directory above the kernel tree, this will be done automagically.

dpkg-buildpackage -S -rfakeroot -I.git -I.gitignore -i'\.git.*' \
       -v<version>

A note to <version>: this should be the last relevant version uploaded. So for uploads to -proposed this should be the last version uploaded to updates. Updates to security should probably be done the same way, but since -security is moved to -updates rather quickly, both will be the same version anyway.

The actual release

The release is done by uploading the source packages to the kernel team ppa. Once it has been built, it get's pocket copied from the ppa to the -proposed pocket. If the kernel ABI has changed, LUM, LBM and LRM must be uploaded with updated version/ABI numbers as well. Also the linux-meta package must be updated and uploaded (after the other packages are done).

Post-release tasks

As described in KernelTeam/KernelMaintenance. Thoughts on that:

  • The ABI files have to be fetched when starting a new release. The maint-startnewrelease script can be used to fetch the ABI files from the builders.

KernelTeam/StableKernelMaintenance (last edited 2016-07-07 19:32:24 by kamalmostafa)