JauntyKernelTreeManagement
Launchpad Entry: jaunty-kernel-tree-mgt
Created: PeteGraner
Contributors: Andy Whitcroft
Packages affected:
Summary
Review of how the kernel team currently manages the git tree, looking for process improvements, automation and things in general to make the kernel developer's day to day life easier. This session will focus on the current kernel tree management, documenting how that management is done and also why it is done that way. Later we will focus on possible improvments to process or the existing automation to simplify the process where possible.
Release Note
This is a process rather than release based Blueprint, and therefore there will be no Release Note update associated with this track.
Rationale
The kernel tree managment is poorly documented and has evolved over time into its current state. The purpose here is to both ensure we understand all of the steps involved in maintaining the kernel tree and also the reasons for those steps. We also intend to spend time looking at process and automation with a view to simplifying our processes and improve automation.
Assumptions
We assume that the reader is familiar with linux kernels, the mainline kernel development and release process, aware of the mainline stable release process, and has an understanding of the git tools used to manage the kernel trees.
Background
The kernel is maintained in the community using git. Linus has the master tree containing the combined development trees.
Linus' Tree
The source for the kernel for our releases is Linus' tree. This is the main development branch. New releases are generated by merging the changes from a large number of subsystem maintainer trees. Blessed combinations are released from here, both pre-releases for testing and primary point releases. These releases form the back bone of the early kernels during our alpha phases.
Stable Trees
After release of a particular point release development moves on with new feastures merging to mainline. The stable team then takes ownership of identifying and back porting critical fixes to the now closed releases. These appear in the 2.6.x.y stable trees. These are a significant source of stable updates for our releases.
Ubuntu's Trees
The Ubuntu kernel aims to be as close as possible to the mainline kernel trees. There are two main reasons to differ from the mainline kernel:
- to handle our Ubuntu packing system (debian/*)
- to pull down bug fixes and functionality which are not yet available in the mainline kernel
We maintain a set of trees per release, named for the release in the kernel.ubuntu.org git repository, for example the Intrepid trees are in the ubuntu/ubuntu-intrepid.git repository, at the URL below:
http://kernel.ubuntu.com/git/
Our trees are actually split into three or four parts depending on the release:
- Linux -- the main tree consisting of the bulk of the kernel,
- Linux Ubuntu Modules -- representing Ubuntu supported modules,
- Linux Backport Modules -- representing useful updated module, and
- Linux Restricted Modules -- representing binary drivers.
Main Tree
The main kernel tree is located in the ubuntu-<release>.git git repository. This contains the primary source for the kernel, plus all of the debian build support, kernel configurations etc required to build the main kernel image packages.
This is based of a specific upstream linux release such as 2.6.27, any stable updates for that release, and any local Ubuntu modifications (Sauce patches).
Linux Ubuntu Modules
The Linux Ubuntu Modules tree carries any additional modules which are not part of mainline which we are supplying and maintaining. In releases prior to Intrepid this was a separate package located in the ubuntu-<release>-lum.git git repository. From Intrepid these are now carried in our Main kernel tree in the ubuntu/ subdirectory.
Linux Backport Modules
The Linux Backport Modules tree carries updated modules and drivers from the latest mainline, it is located in the ubuntu-<release>-lbm.git git repository. Where newer versions of drivers etc appear upstream and those provide significant advantage for for the user base it is desirable to backport those to the main kernel. In some cases the existing driver is functional for a significant existing user base, in this case updating the driver introduces significant risk to the existing user base. In this case we may add the updated driver to the linux backports module allowing those users to opt-in to the new driver.
Linux Restricted Modules
The Linux Restricted Modules tree carries all binary only drivers which are not acceptable into the mainline trees. This does not have a git repository associated.
Release Milestones
In large part kernel updates and releases are driven by the Release Process. A new release cycle starts as soon as the previous release goes final. Preparation for Jaunty started as soon as the 8.10 release was published.
In the following six months a number of alpha and beta releases are made, there are 6 scheduled alphaN snapshots, a beta snapshot, then a release candidate and the final release. Kernels are prepared for each of these releases, typically by rebasing against the latest mainline. Following the final release the kernel is effectivly closed and may only be updated via the SRU process. For longer term support (LTS) releases scheduled updates will also have kernels prepared, but these will not be rebased.
What we do
Starting a new release cycle
For each new release cycle a likely kernel release is picked. This will be the mainline kernel release which should be available in time for the proposed release date. Typically this kernel will be in pre-release (-rcN) for the early alphas before officially releasing.
We will then pull foward all of our Ubuntu specific changes onto this new mainline baseline. During this we review each patch dropping those which are no longer applicable. Once this is complete the repository is pushed up to kernel.ubuntu.com and an Upload performed.
Generating a new repository
The first step is to generate an initial ubuntu/ubuntu-<release>.git, on the central git server zinc.canonical.com:
cd /srv/kernel.ubuntu.com/git/ubuntu GIT_DIR=ubuntu-jaunty.git git init --shared=group cd ubuntu-jaunty.git chmod +x hooks/post-update # Ensure this repository is accessible over http:// git config receive.denynonfastforwards false # Allow forced branch updates initially
NOTE: as we have marked this as a shared=group repository we should not need to set our umask, and so should not need the pre-receive hook.
In order to prevent the first upload being truly enormous we can seed the repository with the majority of Linus' tree from the previous release.
git fetch ../ubuntu-intrepid.git tag v2.6.27 git branch -f master v2.6.27
Clone this repository locally
Then we can clone this repository locally, this can be done one of two ways to reduce the size of the initial download. If you have a local clone of Linus' tree which you use as a common object store you can clone using that as a reference:
git clone --reference linux-linus ssh://zinc.canonical.com/srv/kernel.ubuntu.com/git/ubuntu/ubuntu-jaunty.git
If you prefer to have a fully loaded repository you can still seed that repository as we did on zinc:
mkdir ubuntu-jaunty cd ubuntu-jaunty git init git fetch ../ubuntu-intrepid tag v2.6.27 git branch -f master v2.6.27 git checkout master git remote add origin ssh://zinc.canonical.com/srv/kernel.ubuntu.com/git/ubuntu/ubuntu-jaunty.git git fetch origin
Initial Release Rebase
First we need to pull in the new base tree from Linus' tree (into our local copy):
git remote add linus git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git git fetch linus tag v2.6.28-rc6
We also will need the latest and greatest tree from the previous release:
git fetch --no-tags ../ubuntu-intrepid master:intrepid
We can now review and rebase the changes on the intrepid branch over to the new master branch:
git reset --hard intrepid git rebase -i v2.6.28-rc6 [lots of hard graft]
Once this is complete we are ready to publish this version of the repository:
git push origin master
QUESTIONS:
- what do we do with debian/* changes which occured throughout the release
- likely they are all squashed
- do we actually want this history?
- how might we make this process less difficult
- what is the proceedure for documenting these patches as they roll forward
- where if anywhere is the justification for these patches recorded
- likely the mailing list archive is the official record
Pre-release Rebases
During the early alpha/beta phase we may well be tracking a pre-release kernel tree. That is are based of a -rcN release and wish to track forward towards as newer -rcN releases are made. These are normally performed as a rebase in a similar manner to the Initial Release Rebase:
git fetch linux tag v2.6.28-rc7 git rebase -i v2.6.28-rc7
When the upstream version is finally released (say 2.6.28) we then start using the Upstream Stable Update proceedure below to bring in any updates as they release.
QUESTIONS:
- why are these updates done as a rebase
- we might consider doing these as a merge instead, so that we only go through the rebase pain once at release.
Patching
Patches come in three main forms:
- Ubuntu specific -- eg. changes to the changelog or packaging,
- Ubuntu Sauce -- changes to upstream code but which is not likely to be merged, and
- Upstream -- changes which are already upstream but which are urgent enough to apply locally.
Approvals Process
Emails to the kernel team list.
QUESTIONS:
- could we use patch thingy from oslabs to track patches and approvals
Ubuntu Patches
Ubuntu patches represent any changes to the Ubuntu specific parts of the tree. This includes the packaging control and any Ubuntu specific drivers.
Patches should have a title prefixes with UBUNTU:, be signed off by the committer, acked by the approvers, and reference any applicable launchpag bug:
UBUNTU: bibble: a patch to fix everything Bug: #12345 A cool patch indeed Signed-off-by: Author <author@example.net> Signed-off-by: Committer <committer@example.net> Acked-by: Approver <Approver@example.net>
Sauce Patches
Ubuntu Sauce patches represent changes to upstream code. These are generally changes which are not likely to be accepted in the short term but which are sufficiently useful to our user base that it is worth supporting them out of tree ourselves.
Patches should have a title prefixes with UBUNTU: SAUCE:, be signed off by the committer, and acked by the approvers:
UBUNTU: SAUCE: bibble: a patch to fix everything [...]
Upstream Patches
Upstream patches represent changes which have already been committed to mainline but which are of sufficient importance to our user base that it is worth back porting them. These should be expected to drop from the tree on the next release cycle as they should be included in the kernel we move to.
As these patches should be upstream already we should expect to be cherry-picking them directly into our tree. Each must be signed off by the committer, acked by the approvers, they should reference the upstream commit, and any applicable launchpad bug:
bibble: a patch to fix everything Bug: #12345 commit 1234567890123456789012345678901234567890 upstream [...]
Upstream Stable Updates
Once an upstream release has occured the next release opens. Any critical fixes for bugs and regressions in the recently closed release are picked out for special handling. These patches flow to the stable-release team, are vetted and accumulate in the stable branch. These are periodically released as official 'point releases' 2.6.27.7. We normally pull all of those releases into our tree.
This is normally done by cherry-picking each of these commits into our tree:
git remote add stable-2.6.27 git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.27.y.git git fetch stable-2.6.27 tag v2.6.27.7 git log v2.6.27.6..v2.6.27.7 git cherry-pick <sha1>
This might also be done as a rebase:
git checkout master git reset --hard v2.6.27.7 git rebase -i origin/master
QUESTIONS:
- people clearly have automation to do the cherry-pick phase, should we pull this into out devtools repository
- could we do this using a rebase
- hard to record additional sign-offs, though this could be scripted
- could we use a simple merge from the stable release branch into our tree to simplify managment
- we would not have to handle each patch individually
- using merge.summary=1 we would get an itemised list and we could apply signoff and approvals to the merge commit itself
ABI Bumps
The Ubuntu kernel has an ABI 'level' represented by the -N suffix on the version numver, 2.6.27-10. This represents the kernel ABI signature. Kernel updates which change the signature of an exported function or the shape of a type which is exposed in one of those functions causes an ABI bump. Here the ABI number simply increments. This ABI bump signals that any ABI dependant applications or drivers need to take note of the kernel change. Any upload which will change the ABI should be notified as soon as possible to any interested parties via email.
ABI bumps involve updating the ABI number in the debian changelog and regenerating the control files. This must be done in all of the following, and all of them must then be rebuilt and the packages generated uploaded:
- Kernel tree
- linux-backports-modules
- linux-ubuntu-modules (when present)
- linux-restricted-modules
- linux-meta
CONFIRM:
- there is clearly an order in which they must be uploaded
- I think we can upload them in any order, except linux-meta must be last
Preparing an Upload
From a completly clean source tree (git clean is your friend) you should close off the current release and tag it:
debian/rules insertchanges vi debian/changelog # confirm the changelog and version git add debian/changelog git commit -s -m 'UBUNTU: Ubuntu-2.6.27-10.20' git tag -s -m Ubuntu-2.6.27-10.20 Ubuntu-2.6.27-10.20
Then we must build the source package for upload (adding -sa if you have an original source tarball):
dpkg-buildpackage -S -rfakeroot -I.git -I.gitignore -i'\.git.*'
Finally ensure that this release is reflected in our git respository by pushing both the tree and the tag:
git push origin master Ubuntu-2.6.27-10.20
When do we do the following:
- build the changelog
- update the actual ABI files within the repository
- presumably only on an ABI bump
- presumably before the first change after an upload, ie at startnewrelease step
- perform the startnewrelease step
- presumably before first change after an upload
- when does tagging occur?
Security updates
For security updates which simply cannot wait due to their severity we must will make an interim upload for this. Likely this will apply to the kernel currently in -updates but there will already be a newer kernel either in preparation for, or released to, -proposed. In this case we cannot use the master branch as releasing that would release changes which have not yet been vetted, tested, or approved.
In this scenario we would create a security specific branch from the current version in -updates. The current versions in each pocket are visible here https://launchpad.net/ubuntu/+source/linux on launchpad. We would then cherry-pick in the patches, and pass the result for uploading by the security team.
For example to start a security update against intrepid, we find the pockets populated as below:
intrepid current main release 2.6.27-7.14 main security 2.6.27-9.19 main updates 2.6.27-9.19 main proposed 2.6.27-10.20
So we will branch from the release tag for 2.6.27-9.19:
git checkout -b security Ubuntu-2.6.27-9.19 git cherry-pick <sha1>
We would then Prepare an Upload as normal, sending the upload to security for expedited handling.
Once the release is complete we should ensure that the changes are also in the -proposed tree. One way to do this is to cherry-pick these changes onto -proposed. We must also maintain the debian/changelog entry for the interim security release.
Note that ABI and upload number handling here may be complicated as later kernels may already have been released into -proposed since the kernel in -updates was released. Even where there is no ABI bump to consider non-sequential upload numbers may exist in this branch.
CONFIRM
- that we send the upload to security for build and release
- for non urgent updates we use -proposed but there is talk of needing to copy this kernel to -security and preparing an USN
- makes sense as you can only enable -security, ask Keen on process
- make sure this is documented in our processes
Stable Release Updates (SRUs)
In order to apply patches to released kernels it is necessary to justify the change and obtain approval from the kernel-team, it is then applied to the appropriate tree as per Patching above. When sufficient bugs have been backported a new kernel package is built and uploaded, and formally proposed for SRU into that release.
QUESTIONS:
- I believe that this process gets the kernel as far as -proposed, what then gets it into -updates
Point releases
LTS releases have stable release points at regular fixed intervals. We endevour to only release kernels and particularly ABI bumps at these stable release points. The expected point release dates are defined by the release milestones https://bugs.launchpad.net/ubuntu/+milestones on launchpad.
QUESTIONS:
- should we do SRUs on regular releases to more of a formal timescale
- this might simplify life for out of tree driver maintainers
End-of-life'ing releases
When a release reaches its end-of-life no further development will occur on the kernel. The kernel git tree is archived, moving it into ubuntu-archive on the repository server.
CONFIRM:
- is there anything else we do other than announce it at a distro level?
QUESTIONS:
- can we merge the archived releases as branches in a single history git repository
- there is an enormous amount of common objects so they should merge cheaply
Unresolved issues
Our focus is on improving our implementation of the existing kernel team kernel management process. That process itself should also be reviewed at the same time.
Ideas
- can we merge our archived kernels, they are one tree really
- use merge for stable updates
- backports as smaller packages
- backports use DKMS as framework?
- think about handling of an upstream kernel module
BoF agenda and discussion
pgraner's notes
- New release tasks
- Patch review
- Need to be sure that when we drop a patch, it's an explicit decision, and coordinated with developers who might be depending on it.
- next rebase to include a public review of all patches and their fate (e.g. on ubuntu-devel@lists)?
- Upstreaming of Sauce patches.
- Patch review
- Tree Mgt.
- Locking the tree past kernel freeze
- LP bugs needed for check in
- Why do we rebase?
- Keep our tree in sync with mainline and branch for releases