Dev Week -- Debugging the Kernel -- John Johansen -- Tue, Jul 12th, 2011

   1 [17:01] <jjohansen> Hi, before we start I figured I would introduce myself,  I am John Johansen an Ubuntu contributor and member of the Kernel team
   2 [17:01] <jjohansen> Feel free to ask questions through out in he #ubuntu-classroom-chat channel, and remember to prepend QUESTION: to your question
   3 [17:01] <jjohansen> So lets get started
   4 [17:01] <jjohansen> ------------------------------------------
   5 [17:01] <jjohansen> Debugging Kernels is a vast topic, and there is lots of documentation out there.  Since we only have an hour I thought I would cover a few thing specific to the Ubuntu kernel.  So perhaps the topic really should have been changed to debugging the Ubuntu kernel.  I am not going to walk through a live example as we don't have time for multiple kernel builds, installing and testing, working with the kernel takes time.
   6 [17:02] <jjohansen> First up is the all important documentation link
   7 [17:02] <jjohansen> It has a lot of useful information buried in its links.
   8 [17:02] <jjohansen> it takes a while to read through but its really worth doing if you are interested in the kernel
   9 [17:02] <jjohansen> The Ubuntu kernels are available from git://<release>.git
  10 [17:02] <jjohansen> ie. for natty you would do
  11 [17:02] <jjohansen>   git clone git://
  12 [17:03] <jjohansen>
  13 [17:03] <jjohansen> gives the full details if you need more
  14 [17:03] <jjohansen> The Ubuntu kernel uses debian packaging and fakeroot to build, so you will need to pull in some dependencies
  15 [17:03] <jjohansen>   sudo apt-get build-dep linux-image-$(uname -r)
  16 [17:04] <jjohansen> once you have the kernel you can change directory into the tree and build it
  17 [17:04] <jjohansen>   fakeroot debian/rules clean
  18 [17:04] <jjohansen>   fakeroot debian/rules binary-headers binary-generic
  19 [17:04] <jjohansen> Note, a couple of things, 1. the clean must be done before your first attempt at building, its sets up some of the environment.  2 we are building the one kernel flavour in the above example, and the end result should be some debs.
  20 [17:04] <jjohansen> also if you ever see me use fdr its an alias I set up for fakeroot debian/rules, so I can get a way with doing
  21 [17:04] <jjohansen>   fdr clean
  22 [17:04] <jjohansen> it saves on typing, I'll try not to slip up but just incase ...
  23 [17:04] <jjohansen> Everyone good so far?
  24 [17:05] <jjohansen> Alright onto bisecting
  25 [17:05] <jjohansen> We need to cover a little more information about how the Ubuntu kernel is put together.  Each release of the Ubuntu linux kernel is based on some version of the upstream kernel.  The Ubuntu kernel carries a set of patches on top of the upstream linux kernel.  The patches really can be broken down into 3 catagories, packaging/build and configs, features/drivers that are not upstream (see the ubuntu directory for most of t
  26 [17:05] <jjohansen> During the development cycle, the Ubuntu kernel is rebased on top of the current linux kernel, as the linux kernel is updated so is the development Ubuntu kernel; it occasionally gets rebased on top of newly imported linux kernels.  This means a couple of things patches that have been merged upstream get dropped, and that the Ubuntu patches stay at the top of the stack.
  27 [17:06] <jjohansen> During the development cycle we hit a point called kernel freeze, where we stop rebasing on upstream, and from this point forward only bug fixes are taken with commits going on top.
  28 [17:06] <jjohansen> So why mention all of this?  Because it greatly affects how we can do kernel bisecting.  If a regression occurs after a kernel is released (say the natty kernel), and there is a known good natty kernel, then bisecting is relatively easy.  We can just checkout the natty kernel and start a git bisect between the released kernel tags.
  29 [17:06] <jjohansen> However bisecting between development releases or different released versions (say maverick and natty) of the kernel becomes much more difficult.  This is because there is no continuity because of the rebasing, so bisect doesn't work correctly, and if you are lucky and have continuity the bisecting may remove the packaging patches.
  30 [17:07] <jjohansen> So how do you bisect bugs in the Ubuntu kernel then?  We use the upstream kernel of course :)
  31 [17:07] <jjohansen> There are two ways to do this, the standard upstream build route and using a trick to get debian packages.
  32 [17:07] <jjohansen> The upstream route is good if you are just doing local builds and not distributing the kernels
  33 [17:08] <jjohansen> but if you want other people to install your kernels you are probably best of using the debian packaging route
  34 [17:08] <jjohansen> So the trick to get debian packaging is pretty simple
  35 [17:09] <jjohansen> you checkout the upstream kernel
  36 [17:09] <jjohansen> checkout an Ubuntu kernel
  37 [17:09] <jjohansen> identify a good and bad upstream kernel
  38 [17:10] <jjohansen> you can do this by using the ubuntu mainline kernel builds available from the kernel team ppa
  39 [17:11] <jjohansen>
  40 [17:11] <jjohansen> that saves you from having to build kernel
  41 [17:12] <jjohansen> now copy the debian/ and debian.master/ directories from the ubuntu kernel into the upstream kernel
  42 [17:12] <jjohansen> you do not want to commit these
  43 [17:12] <jjohansen> as that will just make them disappear with the bisect
  44 [17:12] <jjohansen> you can the change directory into the upstream kernel
  45 [17:12] <jjohansen> edit debian.master/changelog, the top of the file should be something like
  46 [17:12] <jjohansen>   linux (2.6.38-10.44) natty-proposed; urgency=low
  47 [17:14] <jjohansen> you want to change the version to something that will have meaning to you
  48 [17:14] <jjohansen> and that can be easily replaced by newer kernels
  49 [17:14] <jjohansen> you use a debian packaging trick to do this
  50 [17:15] <jjohansen> change 2.6.38-10.44 to something like 2.6.38-10.44~jjLP645123.1
  51 [17:16] <jjohansen> the jj indicates me, then the launchpad bug number and I like to use a .X to indicate how far into the bisect
  52 [17:16] <jjohansen> you can use what ever make sense to you
  53 [17:17] <jjohansen> the important part is the ~ which allows kernels with higher version numbers to install over the bisect kernel without breaking things
  54 [17:18] <jjohansen> if you are going to upload the kernel to a ppa you will also want to update the release info
  55 [17:18] <jjohansen> ie. natty-proposed in this example
  56 [17:19] <jjohansen> if you are using the current dev kernel it will say UNRELEASED and you need to specify a release pocket
  57 [17:19] <jjohansen> ie. natty, maverick, ...
  58 [17:19] <jjohansen> however ppa builds are slow and I just about never use them, at least not for regular bug testing
  59 [17:20] <jjohansen> now you can build the kernel
  60 [17:20] <jjohansen>   fakeroot debian/rules clean
  61 [17:20] <jjohansen>   fakeroot debian/rules binary-headers binary-generic
  62 [17:21] <jjohansen> this will churn through and should build some .debs that can be installed using
  63 [17:21] <jjohansen>   dpkg -i
  64 [17:21] <jjohansen> now on to doing the actual bisect
  65 [17:23] <jjohansen> so bisecting is basically a binary search, start with a known good point and bad, cut the commits in half, build a kernel test if its good, rinse lather, and repeat
  66 [17:23] <jjohansen> git bisect is just a tool to help you do this
  67 [17:24] <jjohansen> it is smarter than just doing the cut in half, it actually takes merges and other things into account
  68 [17:24] <jjohansen> the one important thing to note, for these bisects is if you look at the git log etc, you may find yourself in kernel versions outside of your bisect range
  69 [17:25] <jjohansen> this is because of how merges are handled, don't worry about it, git bisect will handle it for you
  70 [17:25] <jjohansen> so the basics of git bisect are
  71 [17:26] <jjohansen> git bisect start <bad> <good>
  72 [17:26] <jjohansen> where bad is the bad kernel and good is the good kernel
  73 [17:26] <jjohansen> the problem is how do you know which kernel versions to use
  74 [17:27] <jjohansen> if you are using the upstream kernels for a bisect then the ubuntu version tags are not available to you
  75 [17:27] <jjohansen> you need to use either a commit sha, or tag in the upstream tree
  76 [17:29] <jjohansen> sorry lost my internet there for a minute
  77 [17:31] <jjohansen> if you used the mainline kernel builds to find a good and bad point then you can just use the kernel tags for those, ie. v2.6.36
  78 [17:31] <jjohansen> if you used and ubuntu kernel then you can find a mapping of kernel versions here
  79 [17:32] <jjohansen>
  80 [17:32] <jjohansen> alright so I was asked what the debian.master directory is about
  81 [17:32] <jjohansen> in the Ubuntu kernel we have two directories to handle the debian packaging
  82 [17:32] <jjohansen> debian and debian.master/
  83 [17:33] <jjohansen> this allows abstracting out parts of the packaging
  84 [17:34] <jjohansen> when you setup a build, the parts of debian.master/ are copied into debian/ and that is used
  85 [17:35] <jjohansen> the difference between the two isn't terribly important for most people, think of debian as the working directory for the packaging, and master as the reference
  86 [17:36] <jjohansen> when I had you edit debian.master/changelog above I could have changed things around and had you edit debian/changelog
  87 [17:36] <jjohansen> however
  88 [17:36] <jjohansen> fakeroot debian/rules clean
  89 [17:36] <jjohansen> will endup copying debian.master/changelog into debian/
  90 [17:37] <jjohansen> thus if you change debian/change log you have to do a full edit on it every time you do a clean
  91 [17:37] <jjohansen> so if you are editing debian/ you do
  92 [17:37] <jjohansen>   fdr clean
  93 [17:37] <jjohansen>   edit debian/changelog
  94 [17:38] <jjohansen> which is the reversion of doing it to debian.master/changelog
  95 [17:38] <jjohansen>   edit debian.master/changelog
  96 [17:38] <jjohansen>   fdr clean
  97 [17:38] <jjohansen> for me editing debian.master/changelog keeps me from making a mistake and building a kernel with out my edits to the kernel version
  98 [17:40] <jjohansen> hopefully that is enough info on the debian/ and debian.master/ for now
  99 [17:40] <jjohansen> and we will jump back to bisecting for a little longer
 100 [17:41] <jjohansen> so assuming you have your kernel version for good and bad you start, your bisection
 101 [17:41] <jjohansen> git will put you on a commit roughly in the middle
 102 [17:41] <jjohansen> then you can do
 103 [17:41] <jjohansen>   fdr clean
 104 [17:42] <jjohansen>   fakeroot debian/rules binary-headers binary-generic
 105 [17:42] <jjohansen> sorry caught myself using fdr
 106 [17:42] <jjohansen> this will build your kernel you can install and test
 107 [17:43] <jjohansen> and then you can input your info into git bisect
 108 [17:43] <jjohansen> ie.
 109 [17:43] <jjohansen>   git bisect good
 110 [17:43] <jjohansen> or
 111 [17:43] <jjohansen>   git bisect bad
 112 [17:43] <jjohansen> the import thing to remember is to not, commit any of your changes to git
 113 [17:44] <jjohansen> I tend to edit the debian.master/changelog and update the .# at the end of my version string every iteration of the bisection
 114 [17:44] <jjohansen> you don't have to do this
 115 [17:45] <jjohansen> you can get away with just rebuilding, straight or if you want doing a partial build
 116 [17:46] <jjohansen> the partial build is a nice trick if you don't have a monster build machine but it doesn't save you anything early on in the bisect, when git is jumping lots of commits and lots of files are getting updated
 117 [17:46] <jjohansen> the trick to doing a partial build in the Ubuntu build system is removing the stamp file
 118 [17:47] <jjohansen> when the kernel is built there are some stamp files generated and placed in
 119 [17:47] <jjohansen>   debian/stamps/
 120 [17:47] <jjohansen> there is one for prepare and one for the actual build
 121 [17:48] <jjohansen> if you build a kernel, and the build stamp file is around, starting a new build will just use the build that already exists and package it into a .deb
 122 [17:48] <jjohansen> you don't want to do this
 123 [17:49] <jjohansen> so after you have stepped you git bisect (git bisect good/bad)
 124 [17:49] <jjohansen> you
 125 [17:49] <jjohansen>   rm debian/stamps/stamp-build-generic
 126 [17:50] <jjohansen> this will cause the build system to try building the kernel again, and make will use its timestamp dependencies to determine what needs to get rebuilt
 127 [17:51] <jjohansen> if the bisect is only stepping within a driver or subsystem this can save you a log of time on your builds, however if the bisect updates lots of files (moves lots of commits) or updates some common includes, you are going to end up doing a full kernel build
 128 [17:52] <jjohansen> so now for the other tack, what do you do if you don't want to mess with the .deb build and just want to build the kernel old fashioned way
 129 [17:52] <jjohansen> well you build as you are familiar with.
 130 [17:52] <jjohansen>   make
 131 [17:52] <jjohansen>   make install
 132 [17:52] <jjohansen>   make modules_install
 133 [17:53] <jjohansen> then you need to create a ramdisk, and update grub
 134 [17:54] <jjohansen>   sudo update-initramfs -c k <kernelversion>
 135 [17:54] <jjohansen> will create the ram disk you need if you don't want to mess with the kernel version
 136 [17:54] <jjohansen>   sudo update-initramfs -c -k all
 137 [17:54] <jjohansen> then you can do
 138 [17:55] <jjohansen>   sudo update-grub
 139 [17:55] <jjohansen> and you are done
 140 [17:55] <jjohansen> so QUESTION: After rm debian/stamps/stamp-build-generic do you still do a fakeroot debian/rules clean when doing the incremental build?
 141 [17:55] <jjohansen> the answer is no
 142 [17:56] <jjohansen> doing a clean will remove all the stamp files, and remove your .o files which will cause a full build to happen
 143 [17:57] <jjohansen> so with only a couple minutes left I am not going to jump into a new topic but will mention something I neglected about about the Ubuntu debian builds
 144 [17:58] <jjohansen> our build system has some extra checks for expected abi, configs, and modules
 145 [17:58] <jjohansen> when building against an upstream kernel you will want to turn these off
 146 [17:58] <jjohansen> you can do this by setting some variables on the command line
 147 [17:58] <jjohansen>   fakeroot debian/rules binary-headers binary-generic
 148 [17:58] <jjohansen> becomes
 149 [17:59] <jjohansen>   skipabi=true skipconfig=true skipmodule=true fakeroot debian/rules binary-headers binary-generic
 150 [17:59] <jjohansen> this can also be used when tuning your own configs etc/
 151 [18:00] <jjohansen> I think I will stop there
 152 [18:00] <jjohansen> thanks for attending, drop by #ubuntu-kernel if you have any questions

MeetingLogs/devweek1107/KernelDebugging (last edited 2011-07-14 08:06:13 by dholbach)