A day in the life of a fixed bug

This page is the condensation of notes and discussion that took place during a kernel developer training. The bug is real and we assume that none of us have a clue (yet) on how to process bugs or add new code in the Canonical/Ubuntu way. We do assume we all know how to use editors, fix and build kernels, and boot machines.

  • Warning /!\ The various examples on this page are from a number of bugs. Not to worry. The process is the same even if the details are not.

We have a bug and its fix. In this case, it is a gfs2 bug that we will fix in the Hardy kernel. It was chosen because the patch already exists. Our job is to go through the steps to get it on its way into a proposed kernel and eventually into the release.

We check upstream commit by way of the link (reference) in Launchpad. The commit itself is not in the Linus tree but it is in the maintainer's tree (linux/kernel/steve/gfs2-2.6). The patch is not the same (it is a backport to the hardy kernel gfs2 version) but the details of the patch itself are not that important. What we will do is add the patch by hand and test it.

Start with the Source

This bug is reported in the Hardy kernel so we need to start with the latest hardy kernel tree, from the hardy repository on If you do not yet have one you will need to clone the Hardy repository, otherwise you will want to ensure your repository is up to date.

Cloning a new repository

The simple way is to simply clone the Hardy repository.

git clone git:// 

This pulls down the whole hardy tree which takes a while and takes up a lot of space that is not really needed for this work. See KernelGitGuide for more detail. One optimisation is to keep a current local clone of Linus's tree and use that as a seed for the clone, allowing git to only clone the difference between the two, saving both download time and space.

git clone --reference linux-linus git:// 
  • Warning /!\ Note the --reference which in this case points to "linux-linus" which is a path to the local clone of Linus' git tree, here at the same level as the git we are about to create. This Linus tree is always kept current and is read-only (by convention). This is a git "good practice".

Updating an existing repository

If you already have a hardy repository you will need to ensure it is up to date with all changes in the main repository. To do this simply change directory into the repository and request a fetch (here we assume that this tree was cloned directly from the main repository as detailed in the previous section):

git fetch origin; git rebase origin

Create a bug specific branch

As we have no control over the order in which patches will be accepted into the main tree, it is helpful if all bugs are applied relative to the current development tip. The easiest way to do this is to work on bug specific branches within your repository. We can now create a branch named after our bug to hold our patch. Branches are cheap and this will isolate what we do to just the patch.

git checkout -b lp276641 origin/master

Apply the Patch

Patches can come from a variety of sources. The first obvious source is via changes you develop with your trusty editor. The other two sources of changes are changes from the upstream git trees or a patch from another source such as the bug reporter or a debdiff file.

A Patch from an Upstream Git

The most common and preferred source of patches (from a release managment point of view) is the upstream. The patch has already been developed and reviewed and as the Ubuntu releases rebase to the upstream, these back-ported patches can be replaced by the mainline submissions.

The first method is to use the tools more commonly used to forward a patch to upstream. Use a two step process to merge the patch candidate. The first step starts in the source git repository. The task here is to extract the patch(es) in a form that preserves the history.

git format-patch <revision>

The revision, usually just the commit id or range of ids, identifies your patch candidate. This will create a file, usually in the top level of the repository that you can then email or copy, if necessary, to the system where your branch is located. See the git documentation for other options that you can use when formatting a patch.

The next step is to apply it to your patch branch. Very simple patches will obviously just drop in but more complex patches should be tried first to detect any merge conflicts without modifying any files.

git apply --check <patch-file>

This version of the apply just reports the result of applying the patch without actually changing any files. If the check fails, you will have to manually resolve the patch. For this discussion, we assume that the result is a clean patch application that we can now apply to the branch

git am -s <patch-file> ...

This command applies one or more patches and the -s adds a sign-off line using your identity to the commit.

If you cloned your repository with a reference to the upstream as described above, there is a simpler method, using a cherry pick to do this process. This is a preferred method because you can preserve more history which is always good, particularly at next-release or rebase time. The key to making this simple method work is that the cherry pick can follow the reference to the upstream to find the patch. All you need is the commit id (SHA1 name) for the patch(es). The following example pulls in two patches in order which is often the case when the upstream has some closely related commits in the same files or when it took multiple commits to finally fix the problem.

git cherry-pick -e -s -x -m 8453cbe81a2f266a5a6724a613be9328cff89187 \

The -s applies your signoff, the -e drops you into your favorite editor to add additional comments, and, the -x adds a very important line indicating the upstream commit from which the patch came.

A Patch from a non-Git Source

In our example the upstream patch is a performance optimization. The patch we find in launchpad is a fix on top of it. Follow the link in comment #8 to the patch.

It started as a debdiff. We will take just the patch itself from the attachment in the launchpad comment and apply it by hand.

patch -p1 <~/PATCH

If the patch updates debian/changelog you will need to undo any changes there. The changelog is generated automatically at release time from the git commit history. Use git checkout debian/changelog will undo any changes here.

Test Build the patch

  • Warning /!\ Assume we added alias fdr='fakeroot debian/rules' into our ~/.bashrc to make things a bit easier to read.

The first step is to prepare the build if this git tree is fresh.

fdr prepare-generic

We can then build one flavor of the kernel package to see if it works.

fdr binary-debs flavours=generic   #note the .uk spelling.

A second way is to just build the portion of the tree that we applied the patch to. We can try this here because the patch is simple and has already been tested. We just want make sure that the patch applied cleanly.

make -C debian/build/build-generic M=`pwd`/fs/gfs2

This approach should only be used in known circumstances. If there is any chance that that the new tree may have new dependencies, use debuild -b to clean up the tree before doing a complete build. This will check the sanity of the environment and the source pool.

Publish a Package for Testing

There are some, very few, bugs that can be confirmed as fixed by just looking at the patch. All others need to be tested by the reporter(s) just to be sure because their particular configuration is available to you for testing. The Personal Package Archive (PPA) is the vehicle to publish test packages to the user community for testing.

PPA Setup

Setting up and using your PPA is a multi-step process. For complete detail on how to setup and use a PPA consult PPA Help page. In short, you need to do the following:

  • Activate your PPA archive. This is done by way of your personal page and profile in Launchpad. Follow the links and instructions to turn it on.
  • Activate your GPG key. This is also done from your personal page. We assume here that you have already sent in your public key. This step activates it for the PPA.
  • Become an Ubuntero. Go to your personal page, look toward the bottom and find the Ubuntero line. If it says No,click through and follow the directions. You will be signing a pledge to refrain from social immaturity when working with the community.

At this point, you can follow the links in Launchpad to your PPA and check things out. It will be empty at this point but we are now ready to upload.

Building and Uploading the Package

You may have built binary packages to do local testing but the PPA wants a source archive. The process starts with an upload of a source package which will then be built by the build farm for all architectures supported by the release. The resulting packages are then published to your archive.

The first step is to create a source archive. Starting in the git branch for your bug, edit the debian/changelog. The result should look like:

--- a/debian/changelog
+++ b/debian/changelog
@@ -1,10 +1,10 @@
-linux (2.6.27-10.21) UNRELEASED; urgency=low
+linux (2.6.27-10.21~lp287701lieb1) intrepid; urgency=low

   CHANGELOG: Do not edit directly. Autogenerated at release.
   CHANGELOG: Use the printchanges target to see the curent changes.
   CHANGELOG: Use the insertchanges target to create the final log.

- --  Tim Gardner <>  Mon, 24 Nov 2008 10:50:51 -0700
+ --  Jim Lieb <>  Mon, 24 Nov 2008 10:50:51 -0700

There are two lines of interest. Edit the first line to change the package version and release.

  • ~lp287701lieb1 - This is appended to the package name and release to separately name this package in an apt friendly way. The ~ separates the release identifier from your sub-release. Note the bug id and your (my) name in the sub-release. This identifies the package with the bug and anticipates more than one try by more than one developer to stomp this bug down.

  • intrepid - This flags the release. Note that the PPA engine does not know about UNRELEASED or -proposed releases.

Edit the "change by" line to refer to yourself.

The next step is to prepare the source package for upload:

dpkg-buildpackage -S -rfakeroot -I.git -I.gitignore -i'\.git.*' -k08B32ADF793FC146

Consult the man page for the build options. The -k option should refer to the same GPG key you activated for the PPA. Without it, Launchpad will silently drop your bits on the floor even though your key may already be in the company key server. The end result of this command is three files in the parent directory, a .changes file, a .tar.gz archive, and a .dsc file.

The next step is to upload the package. Connect to the parent directory and send the package to Launchpad. To do this, we assume you have one of dput or dupload installed.


You will need a personal configuration for dput in your home directory ~/ which should look like:

fqdn =
method = ftp
incoming = ~lieb/ubuntu
login = anonymous
allow_unsigned_uploads = 0

The command to upload to the Launchpad PPA server is:

dput lp-ppa linux_2.6.27-10.21~lp287701lieb1_source.changes


You will need a personal configuration for dupload in ~/.dupload.conf something like the following:

package config;
$default_host = "my-ppa";

$cfg{'my-ppa'} = {
        fqdn => "",
        method => "ftp",
        login => "anonymous",
        incoming => "~apw/ubuntu",
        visibleuser => "apw",
        visiblename => "",
        passive => 1,

To upload your source use the following command (an optional --to <destination> selects an alternative upload site):

dupload linux_2.6.27-10.21~lp287701lieb1_source.changes

upload results

Note that we pass the .changes file as the argument and dput uses that to figure out the other two files. You will receive an email to your Launchpad email address when the upload completes. If email reports that your upload failed, add a -f option to the command line on your next attempt(s). Otherwise, dput is convinced that it is already uploaded.

At some point, your packages will appear in the archive. Re-visit the PPA page mentioned above to check its status. The page should now report statistics and the build state for your upload. This process may take more than a few hours due to other packages in the queue. The system will automatically build for all of the architectures supported by the release.

Notification and Resolution

Once the packages have been built successfully, you can notify the bug reporter(s) that a test package is available for them to download. Do this by adding a comment to the bug in Launchpad that looks something like this:

<comments about the patch and what to look for etc.>

The packages can be found at:

Follow the directions for updating from a PPA.

Note that this kernel is for testing only. Once this issue is verified, the patch will be forwarded into the update process for intrepid.

The key part beyond the URL to find the package(s) is to remind everybody that this is a test package, nothing more. Once the reporter(s) comment back via the Launchpad bug that things work properly, you can commit the patch.

  • Warning /!\ Don't forget to revert the change to debian/changelog that you made to create the source package!

Commit the Patch

We will now commit the patch using a template from within the git tree (in all later releases):

git commit -a -e -s -F debian/commit-templates/sauce-patch

The -a adds the changes to the index first, the -s adds our signoff, and the -e drops us into our favorite editor. The sauce template comes up in our favorite editor and we edit the fields. Here is what the commit from the sauce template looks like after we edited it in vi:

    UBUNTU: SAUCE: replace gfs2_bitfit with upstream version to prevent oops

    OriginalAuthor: Sergio Tosti <>

    Backport of the recent mainline gfs2_bitfit() to fix occasional OOPS.
    This is already fixed in mainline and does not apply to Intrepid or
    Signed-off-by: Andy Whitcroft <>

Everything in the template with a # at the beginning of the line is stripped automatically on commit so you can just ignore them while editing. These provide hints as to how to fill out the template. The key fields here are the first line, Bug, and Signed-off-by. The OriginalAuthor is included where appropriate, particularly if the patch came from upstream. It is very important that the one line summary (first line) is separated from the remainder to prevent the Ubuntu tag fields being pulled up into the summary.

Filling in the BugLink field is important since it is automatically linked to Launchpad bug (LP: xxxxxx) and the bug status is changed automatically as the released package makes its way into -proposed and -updates. Also should patches arrive in our tree via upstream this link is meaningful to upstream during review, and will automatically link our bugs when we receive via that route.

  • Warning /!\ A word about SAUCE. This tag to commit is used in cases where a patch is added that is not in the upstream. This is both a reminder that a patch is special and should be likely be pushed to upstream, as well as something that can be used to find those patches that need to be taken to a new tree when the next release is started.

Forward the Commit for Approval

The next step to pass the patch on for approval is manual. Use your favorite mail agent to send the email to the kernel-team list to have it reviewed:

First we generate the patch and its diff.

git format-patch HEAD~1..HEAD

We have just the one commit so this captures it into a file named "0001-<the first line of the commit>.patch". We now send it with

git send-email \
        --from 'Andy Whitcroft <>' \
        --identity canonical \
        --no-chain-reply-to \
        --thread \
        --suppress-cc all \
        --to \
        --compose \

The "????-*" is our patch file. The editor comes up and we enter the introduction of our submittal. This is what it looks like just before we send it.

From: Andy Whitcroft <>
Subject: [PATCH 0/1] SRU BUG #276641 -- fix gfs2_bitfit OOPS

Hardy is vunerable to a gfs2 OOPS when copying large files.  I have built
kernels with this patch applied.  The submitter has tested it fixes the
issue.  This bug is already fixed in mainline, Intrepid is already immune.

Following this email is the patch as attached to the bug.

Proposing this for SRU to Hardy.


The diffstat would be included by the script. In this case, the patch itself is in email 1/1. This may be an iterative process as we address issues that other developers may raise.

Apply approvals

Before we push the change, we must include the approvals we received from other developers on kernel-team. Open the commit and add the approvals after the Signed-off-by line in the commit we did earlier.

git commit --amend

This will change the commit name but a git show will display it for us to cut-and-paste to the submission email. The patch is now ready for the main repository.

Push the Change

If you are reading this page and going through the process for the first time you probably do not have access to push your change directly to the master repository. Therefore we must publish the change to a personal repository on and then request the change be pulled into the main tree. This is where the branch-per-patch becomes useful.

Setting your local umask

Before pushing to any repositories, you must make sure that your local umask is set to 0002. Add 'umask 0002' to your local .bashrc file at the top like this, before the exit for non-interactive shells:

umask 0002
# If not running interactively, don't do anything

Personal repository setup

Before we actually do the push, we need a repository to push to. Since we assumed that we have not done this before, we have to create it first. To do that, we log into, and create our own repository as a bare clone of the master.

# we got here by way of ssh...
cd /srv/<your-login>  # created for us by IT...
git clone --bare ../ubuntu/ubuntu-hardy.git/ ubuntu-hardy.git

This results in a repository that we can push our changes to although it cannot be used for anything else. The next step is to link the local repository to it so we can push the change without breaking our back. Add a remote called up with

git remote add up git://<your-login><your-login>/<repository>.git

This entry in .git/config should now resemble the following. Remember that your-login and repository need to be replaced appropriately.

[remote "up"]
        url = git+ssh://
        fetch = +refs/heads/*:refs/remotes/origin/*

Note the git+ssh: form, this indicates that we will ssh transport for this remote, and it is this which makes this remote writable. This means we can push new branches to this remote as well as fetching from it.

Pushing the Change

We now push it up to our repository on If we added that repository as up in our repository config, we can just push the branch as below:

git push up +lp276641
  • Warning /!\ Note the + in +lp276641. This will overwrite the branch! We may have pushed the branch earlier so that others could access the patch for testing and this will update the branch to the final commit. DO NOT do this to the master repository!

Requesting a pull

The last step is to send an email to kernel-team requesting a pull of the change. The subject line should contain:

  • SRU LP <launchpad bug number> <summary line from bug.

and the body should contain the following contents although it can be free-form:

  • A link to the Launchpad bug, i.e.

  • A link to the repository where the patch can be found, and
  • The commit name.

The following is an example body from another bug:

The patch with updated approvals and commit message is available at:

git:// lp292429

The commit is:

The appropriate person on the kernel team will pull the patch from the named repository and commit it to the master repository.

The SRU Sequence

At this point in the process flow, the patch is ready to be nominated for inclusion into the release or update. The next step is to nominate the patch for inclusion into the next SRU. The commit message in the repository or the email traffic on are not enough.

This nomination/notification is done on Launchpad after you have received a reply from your pull request confirming that your change has been pulled and committed to the mainline or after you have made the commit yourself if you have push access to the mainline tree. Open the bug and add a bug comment that contains a justification for the nomination as well as links to identify the patch. The following is an example comment from another bug:

SRU Justification

Impact: Add support for an infra-red remote controller.

Patch Description: Backported from commit ba340b40a5f65261731583f67d7ec8cafbf5cfaa upstream.


Test Case: See bug description

See StableReleaseUpdates for more details on the release process.

Finally when the update is finally released to the public the Launchpad Janitor will move the bug to Fix Released indicating the fix is now available to all.

Kernel/Dev/KernelBugFixing (last edited 2012-12-13 21:36:28 by chiluk)