Launchpad Entry: jaunty-kernel-tree-mgt
Contributors: Andy Whitcroft
Review of how the kernel team currently manages the git tree, looking for process improvements, automation and things in general to make the kernel developer's day to day life easier. This session will focus on the current kernel tree management, documenting how that management is done and also why it is done that way. Later we will focus on possible improvments to process or the existing automation to simplify the process where possible.
This is a process rather than release based Blueprint, and therefore there will be no Release Note update associated with this track.
The kernel tree managment is poorly documented and has evolved over time into its current state. The purpose here is to both ensure we understand all of the steps involved in maintaining the kernel tree and also the reasons for those steps. We also intend to spend time looking at process and automation with a view to simplifying our processes and improve automation.
We assume that the reader is familiar with linux kernels, the mainline kernel development and release process, aware of the mainline stable release process, and has an understanding of the git tools used to manage the kernel trees.
The kernel is maintained in the community using git. Linus has the master tree containing the combined development trees.
The source for the kernel for our releases is Linus' tree. This is the main development branch. New releases are generated by merging the changes from a large number of subsystem maintainer trees. Blessed combinations are released from here, both pre-releases for testing and primary point releases. These releases form the back bone of the early kernels during our alpha phases.
After release of a particular point release development moves on with new feastures merging to mainline. The stable team then takes ownership of identifying and back porting critical fixes to the now closed releases. These appear in the 2.6.x.y stable trees. These are a significant source of stable updates for our releases.
The Ubuntu kernel aims to be as close as possible to the mainline kernel trees. There are three main reasons to differ from the mainline kernel:
- to handle our Ubuntu packing system (debian/*)
- to pull down bug fixes and functionality which are not yet available in the mainline kernel
- to handle distribution specific needs that may not be appropriate for upstream inclusion
We maintain a set of trees per release, named for the release in the kernel.ubuntu.org git repository, for example the Intrepid trees are in the ubuntu/ubuntu-intrepid.git repository, at the URL below:
Our trees are actually split into three or four parts depending on the release:
- Linux -- the main tree consisting of the bulk of the kernel,
- Linux Ubuntu Modules -- representing Ubuntu supported modules,
- Linux Backport Modules -- representing useful updated module, and
- Linux Restricted Modules -- representing binary drivers.
The main kernel tree is located in the ubuntu-<release>.git git repository. This contains the primary source for the kernel, plus all of the debian build support, kernel configurations etc required to build the main kernel image packages.
This is based of a specific upstream linux release such as 2.6.27, any stable updates for that release, and any local Ubuntu modifications (Sauce patches).
Linux Ubuntu Modules
The Linux Ubuntu Modules tree carries any additional modules which are not part of mainline which we are supplying and maintaining. In releases prior to Intrepid this was a separate package located in the ubuntu-<release>-lum.git git repository. From Intrepid these are now carried in our Main kernel tree in the ubuntu/ subdirectory.
Linux Backport Modules
The Linux Backport Modules tree carries updated modules and drivers from the latest mainline, it is located in the ubuntu-<release>-lbm.git git repository. Where newer versions of drivers etc appear upstream and those provide significant advantage for for the user base it is desirable to backport those to the main kernel. In some cases the existing driver is functional for a significant existing user base, in this case updating the driver introduces significant risk to the existing user base. In this case we may add the updated driver to the linux backports module allowing those users to opt-in to the new driver.
Linux Restricted Modules
The Linux Restricted Modules tree carries all binary only drivers which are not acceptable into the mainline trees. This does not have a git repository associated.
In large part kernel updates and releases are driven by the Release Process. A new release cycle starts as soon as the previous release goes final. Preparation for Jaunty started as soon as the 8.10 release was published.
In the following six months a number of alpha and beta releases are made, there are 6 scheduled alphaN snapshots, a beta snapshot, then a release candidate and the final release. Kernels are prepared for each of these releases, typically by rebasing against the latest mainline. Following the final release the kernel is effectivly closed and may only be updated via the SRU process. For longer term support (LTS) releases scheduled updates will also have kernels prepared, but these will not be rebased.
What we do
Starting a new release cycle
For each new release cycle a likely kernel release is picked. This will be the mainline kernel release which should be available in time for the proposed release date. Typically this kernel will be in pre-release (-rcN) for the early alphas before officially releasing.
We will then pull foward all of our Ubuntu specific changes onto this new mainline baseline. During this we review each patch dropping those which are no longer applicable. Once this is complete the repository is pushed up to kernel.ubuntu.com and an Upload performed.
Generating a new repository
The first step is to generate an initial ubuntu/ubuntu-<release>.git, on the central git server zinc.canonical.com:
cd /srv/kernel.ubuntu.com/git/ubuntu GIT_DIR=ubuntu-jaunty.git git init --shared=group cd ubuntu-jaunty.git chmod +x hooks/post-update # Ensure this repository is accessible over http:// git config receive.denynonfastforwards false # Allow forced branch updates initially
NOTE: as we have marked this as a shared=group repository we should not need to set our umask, and so should not need the pre-receive hook.
In order to prevent the first upload being truly enormous we can seed the repository with the majority of Linus' tree from the previous release.
git fetch ../ubuntu-intrepid.git tag v2.6.27 git branch -f master v2.6.27
Clone this repository locally
Then we can clone this repository locally, this can be done one of two ways to reduce the size of the initial download. If you have a local clone of Linus' tree which you use as a common object store you can clone using that as a reference:
git clone --reference linux-linus ssh://zinc.canonical.com/srv/kernel.ubuntu.com/git/ubuntu/ubuntu-jaunty.git
If you prefer to have a fully loaded repository you can still seed that repository as we did on zinc:
mkdir ubuntu-jaunty cd ubuntu-jaunty git init git fetch ../ubuntu-intrepid tag v2.6.27 git branch -f master v2.6.27 git checkout master git remote add origin ssh://zinc.canonical.com/srv/kernel.ubuntu.com/git/ubuntu/ubuntu-jaunty.git git fetch origin
Initial Release Rebase
First we need to pull in the new base tree from Linus' tree (into our local copy):
git remote add linus git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git git fetch linus tag v2.6.28-rc6
We also will need the latest and greatest tree from the previous release:
git fetch --no-tags ../ubuntu-intrepid master:intrepid
We can now review and rebase the changes on the intrepid branch over to the new master branch:
git reset --hard intrepid git rebase -i v2.6.28-rc6 [lots of hard graft]
Once this is complete we are ready to publish this version of the repository:
git push origin master
- what do we do with debian/* changes which occured throughout the release
- likely they are all squashed
- do we actually want this history?
- how might we make this process less difficult
what is the proceedure for reviewing & documenting these patches as they roll forward
- where if anywhere is the justification for these patches recorded
- patch tracking file that is under git control?
- the mailing list archive is the official record? (Can this scale?)
During the early alpha/beta phase we may well be tracking a pre-release kernel tree. That is are based of a -rcN release and wish to track forward towards as newer -rcN releases are made. These are normally performed as a rebase in a similar manner to the Initial Release Rebase:
git fetch linux tag v2.6.28-rc7 git rebase -i v2.6.28-rc7
When the upstream version is finally released (say 2.6.28) we then start using the Upstream Stable Update proceedure below to bring in any updates as they release.
- why are these updates done as a rebase
- we might consider doing these as a merge instead, so that we only go through the rebase pain once at release.
Notes(smb): IIRC we do this because that should make it simpler to send SAUCE patches upstream. The rebase enforces that those apply cleanly to a more recent upstream kernel.
When we reach the main release milestone for a release we start the next release as described in Starting a new release cycle above. At this point the release is effectively frozen and we may only update the release via the Stable Release Updates (SRU) which is described in more detail later.
At this point the git repository is effectively closed for new updates. All updates after this will have been approved via the SRU process and should have an associated launchpad bug number in the commits.
- we probably should refuse non-fast-forward updates at this stage
- git config receive.denynonfastforwards true
- should we also add some additional checks to the commit hooks to enforce LP#s in the commits
Notes(smb): 1. In real life I had cases were I did non-forward pushes to correct things on stages between uploads. To fixup some crud. I know it is evil but was convenient.
On 2.: Ben has created some hooks. They reside in the buildscripts.git repo in git-hooks. These already would cover some stuff. Not sure about the bug #.
Patches come in three main forms:
- Ubuntu specific -- eg. changes to the changelog or packaging,
- Ubuntu Sauce -- changes to upstream code but which have not yet, or are not likely to be merged, and
- Upstream -- changes which are already upstream but which are urgent enough to apply locally.
Every patch being committed to the archive should have been reviewed and approved by members of the kernel team. This is particularly important once the main release is reached. Approval occurs on the kernel-team mailing list and should be recorded in the commits via the Acked-by: annotations.
Ubuntu patches represent any changes to the Ubuntu specific parts of the tree. This includes the packaging control and any Ubuntu specific drivers.
Patches should have a title prefixes with UBUNTU:, be signed off by the committer, acked by the approvers, and reference any applicable launchpag bug:
UBUNTU: bibble: a patch to fix everything Bug: #12345 A cool patch indeed Signed-off-by: Author <firstname.lastname@example.org> Signed-off-by: Committer <email@example.com> Acked-by: Approver <Approver@example.net>
Ubuntu Sauce patches represent changes to upstream code. These are generally changes which are are not yet, or not likely to be accepted upstream in the short term but which are sufficiently useful to our user base that it is worth supporting them out of tree ourselves.
Patches should have a title prefixes with UBUNTU: SAUCE:, be signed off by the committer, and acked by the approvers:
UBUNTU: SAUCE: bibble: a patch to fix everything [...]
These patches should be being pushed upstream to reduce our maintenance burden in the next release cycle. We do not wish to diverge from mainline any more than absolutly necessary.
- how do we document which patches we have pulled forward
- UDS is after our alpha1 deadlines so we may have done this before then
- could we use the patchworks patch tracker at oslabs to track our source patches and approvals
Upstream patches represent changes which have already been committed to mainline but which are of sufficient importance to our user base that it is worth back porting them. These should be expected to drop from the tree on the next release cycle as they should be included in the kernel we move to.
As these patches should be upstream already we should expect to be cherry-picking them directly into our tree. Each must be signed off by the committer, acked by the approvers, they should reference the upstream commit, and any applicable launchpad bug:
bibble: a patch to fix everything Bug: #12345 commit 1234567890123456789012345678901234567890 upstream [...]
Upstream Stable Updates
Once an upstream release has occurred the next release opens. Any critical fixes for bugs and regressions in the recently closed release are picked out for special handling. These patches flow to the stable-release team, are vetted and accumulate in the stable branch. These are periodically released as official 'point releases' 184.108.40.206. We normally pull all of those releases into our tree.
This is normally done by cherry-picking each of these commits into our tree:
git remote add stable-2.6.27 git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.27.y.git git fetch stable-2.6.27 tag v220.127.116.11 git log v18.104.22.168..v22.214.171.124 git cherry-pick <sha1>
This might also be done as a rebase:
git checkout master git reset --hard v126.96.36.199 git rebase -i origin/master
- people clearly have automation to do the cherry-pick phase, should we pull this into out devtools repository
- could we do this using a rebase
- hard to record additional sign-offs, though this could be scripted
- could we use a simple merge from the stable release branch into our tree to simplify managment
- we would not have to handle each patch individually
- using merge.summary=1 we would get an itemized list and we could apply signoff and approvals to the merge commit itself
Notes(smb): 1. Would be a good Idea. I have started some scripts as the others have. Certainly all depend a bit on the agreed way of proceeding.
2. + 3.: I would favor the merge since rebase kills the SHA references. I did not know of merge summary and that probably requires some enhancements to the script that generates the changelog (to have the bug number also on the individual changes).
The Ubuntu kernel has an ABI 'level' represented by the -N suffix on the version numver, 2.6.27-10. This represents the kernel ABI signature. Kernel updates which change the signature of an exported function or the shape of a type which is exposed in one of those functions causes an ABI bump. Here the ABI number simply increments. This ABI bump signals that any ABI dependent applications or drivers need to take note of the kernel change. Any upload which will change the ABI should be notified as soon as possible to any interested parties via email.
ABI bumps involve updating the ABI number in the debian changelog and regenerating the control files. This must be done in all of the following, and all of them must then be rebuilt and the packages generated uploaded:
- Kernel tree
- linux-ubuntu-modules (when present)
- there is clearly an order in which they must be uploaded
- I think we can upload them in any order, except linux-meta must be last
Notes(smb): The order of upload seems irrelevant. The uploads have to be accepted by archive administrators which pick an order. The build order itself should be lum first then lbm. At least for Hardy lum and lbm generate sparate header packages and lbm depends on the lum headers
Preparing an Upload
From a completly clean source tree (git clean is your friend) you should close off the current release and tag it:
debian/rules insertchanges vi debian/changelog # confirm the changelog and version git add debian/changelog git commit -s -m 'UBUNTU: Ubuntu-2.6.27-10.20' git tag -s -m Ubuntu-2.6.27-10.20 Ubuntu-2.6.27-10.20
Then we must build the source package for upload (adding -sa if you have an original source tarball):
dpkg-buildpackage -S -rfakeroot -I.git -I.gitignore -i'\.git.*'
Finally ensure that this release is reflected in our git respository by pushing both the tree and the tag:
git push origin master Ubuntu-2.6.27-10.20
- when do we update the actual ABI files within the repository
- presumably only on an ABI bump
- presumably before the first change after an upload, ie at startnewrelease step
- when do we perform the startnewrelease step
- presumably before first change after an upload
Notes(smb): 1a yes. 1b at least in the past, after the checkin that requires it.
2. yes. Preferably after the previous kernel has been released, so the abi files can be pulled (debian/scripts/misc/getabis <kver> <rel> (2.6.28 1.1)
The ABI files should not be forgotten. Each build checks against files in debian/abi/<kver>-<rel> (e.g. 2.6.28-1.1). This has to be refreshed at least before upload. In the past on stable kernels without abi changes this has been short-circuited by git-mv'ing the directory. The full process would be (as I did it): get the new files with debian/scripts/misc/getabis, tar it and remove the directory again, then git-mv the old dir to the new name and extract the previously created tar. That way unchanged files are recognized as being moved.
For security updates which simply cannot wait due to their severity we must will make an interim upload for this. Likely this will apply to the kernel currently in -updates but there will already be a newer kernel either in preparation for, or released to, -proposed. In this case we cannot use the master branch as releasing that would release changes which have not yet been vetted, tested, or approved.
In this scenario we would create a security specific branch from the current version in -updates. The current versions in each pocket are visible here https://launchpad.net/ubuntu/+source/linux on launchpad. We would then cherry-pick in the patches, and pass the result for uploading by the security team.
For example to start a security update against intrepid, we find the pockets populated as below:
intrepid current main release 2.6.27-7.14 main security 2.6.27-9.19 main updates 2.6.27-9.19 main proposed 2.6.27-10.20
So we will branch from the release tag for 2.6.27-9.19:
git checkout -b security Ubuntu-2.6.27-9.19 git cherry-pick <sha1>
We would then Prepare an Upload as normal, sending the upload to security for expedited handling.
Once the release is complete we should ensure that the changes are also in the -proposed tree. One way to do this is to cherry-pick these changes onto -proposed. We must also maintain the debian/changelog entry for the interim security release.
Note that ABI and upload number handling here may be complicated as later kernels may already have been released into -proposed since the kernel in -updates was released. Even where there is no ABI bump to consider non-sequential upload numbers may exist in this branch.
- that we send the upload to security for build and release
- for non urgent updates we use -proposed but there is talk of needing to copy this kernel to -security and preparing an USN
- makes sense as you can only enable -security, ask Keen on process
- make sure this is documented in our processes
Notes(smb): 1. yes security.canonical.com (aka jackass)
Stable Release Updates (SRUs)
In order to apply patches to released kernels it is necessary to justify the change and obtain approval from the kernel-team, it is then applied to the appropriate tree as per Patching above. When sufficient bugs have been backported a new kernel package is built and uploaded, and formally proposed for SRU into that release.
- I believe that this process gets the kernel as far as -proposed, what then gets it into -updates
Notes(smb): Ask an archive administrator to move it. The kernel should have been in -proposed for at least a week and there must be no open known issues,
LTS releases have stable release points at regular fixed intervals. We endevour to only release kernels and particularly ABI bumps at these stable release points. The expected point release dates are defined by the release milestones https://bugs.launchpad.net/ubuntu/+milestones on launchpad.
- should we do SRUs on regular releases to more of a formal timescale
- this might simplify life for out of tree driver maintainers
When a release reaches its end-of-life no further development will occur on the kernel. The kernel git tree is archived, moving it into ubuntu-archive on the repository server.
- is there anything else we do other than announce it at a distro level?
- can we merge the archived releases as branches in a single history git repository
- there is an enormous amount of common objects so they should merge cheaply
Our focus is on improving our implementation of the existing kernel team kernel management process. That process itself should also be reviewed at the same time.
Other ideas which need some consideration:
- think about how we might handle an automated upstream kernel module
- backports as smaller packages
- backports use DKMS as framework?