== Dev Week -- Debugging the Kernel -- John Johansen -- Tue, Jul 12th, 2011 == {{{#!irc [17:01] Hi, before we start I figured I would introduce myself, I am John Johansen an Ubuntu contributor and member of the Kernel team [17:01] Feel free to ask questions through out in he #ubuntu-classroom-chat channel, and remember to prepend QUESTION: to your question [17:01] So lets get started [17:01] ------------------------------------------ [17:01] Debugging Kernels is a vast topic, and there is lots of documentation out there. Since we only have an hour I thought I would cover a few thing specific to the Ubuntu kernel. So perhaps the topic really should have been changed to debugging the Ubuntu kernel. I am not going to walk through a live example as we don't have time for multiple kernel builds, installing and testing, working with the kernel takes time. [17:02] First up is the all important documentation link https://wiki.ubuntu.com/Kernel [17:02] It has a lot of useful information buried in its links. [17:02] it takes a while to read through but its really worth doing if you are interested in the kernel [17:02] The Ubuntu kernels are available from git://kernel.ubuntu.com/ubuntu/ubuntu-.git [17:02] ie. for natty you would do [17:02] git clone git://kernel.ubuntu.com/ubuntu/ubuntu-natty.git [17:03] https://wiki.ubuntu.com/Kernel/Dev/KernelGitGuide [17:03] gives the full details if you need more [17:03] The Ubuntu kernel uses debian packaging and fakeroot to build, so you will need to pull in some dependencies [17:03] sudo apt-get build-dep linux-image-$(uname -r) [17:04] once you have the kernel you can change directory into the tree and build it [17:04] fakeroot debian/rules clean [17:04] fakeroot debian/rules binary-headers binary-generic [17:04] Note, a couple of things, 1. the clean must be done before your first attempt at building, its sets up some of the environment. 2 we are building the one kernel flavour in the above example, and the end result should be some debs. [17:04] also if you ever see me use fdr its an alias I set up for fakeroot debian/rules, so I can get a way with doing [17:04] fdr clean [17:04] it saves on typing, I'll try not to slip up but just incase ... [17:04] Everyone good so far? [17:05] Alright onto bisecting [17:05] We need to cover a little more information about how the Ubuntu kernel is put together. Each release of the Ubuntu linux kernel is based on some version of the upstream kernel. The Ubuntu kernel carries a set of patches on top of the upstream linux kernel. The patches really can be broken down into 3 catagories, packaging/build and configs, features/drivers that are not upstream (see the ubuntu directory for most of t [17:05] During the development cycle, the Ubuntu kernel is rebased on top of the current linux kernel, as the linux kernel is updated so is the development Ubuntu kernel; it occasionally gets rebased on top of newly imported linux kernels. This means a couple of things patches that have been merged upstream get dropped, and that the Ubuntu patches stay at the top of the stack. [17:06] During the development cycle we hit a point called kernel freeze, where we stop rebasing on upstream, and from this point forward only bug fixes are taken with commits going on top. [17:06] So why mention all of this? Because it greatly affects how we can do kernel bisecting. If a regression occurs after a kernel is released (say the natty kernel), and there is a known good natty kernel, then bisecting is relatively easy. We can just checkout the natty kernel and start a git bisect between the released kernel tags. [17:06] However bisecting between development releases or different released versions (say maverick and natty) of the kernel becomes much more difficult. This is because there is no continuity because of the rebasing, so bisect doesn't work correctly, and if you are lucky and have continuity the bisecting may remove the packaging patches. [17:07] So how do you bisect bugs in the Ubuntu kernel then? We use the upstream kernel of course :) [17:07] There are two ways to do this, the standard upstream build route and using a trick to get debian packages. [17:07] The upstream route is good if you are just doing local builds and not distributing the kernels [17:08] but if you want other people to install your kernels you are probably best of using the debian packaging route [17:08] So the trick to get debian packaging is pretty simple [17:09] you checkout the upstream kernel [17:09] checkout an Ubuntu kernel [17:09] identify a good and bad upstream kernel [17:10] you can do this by using the ubuntu mainline kernel builds available from the kernel team ppa [17:11] http://kernel.ubuntu.com/~kernel-ppa/mainline/ [17:11] that saves you from having to build kernel [17:12] now copy the debian/ and debian.master/ directories from the ubuntu kernel into the upstream kernel [17:12] you do not want to commit these [17:12] as that will just make them disappear with the bisect [17:12] you can the change directory into the upstream kernel [17:12] edit debian.master/changelog, the top of the file should be something like [17:12] linux (2.6.38-10.44) natty-proposed; urgency=low [17:14] you want to change the version to something that will have meaning to you [17:14] and that can be easily replaced by newer kernels [17:14] you use a debian packaging trick to do this [17:15] change 2.6.38-10.44 to something like 2.6.38-10.44~jjLP645123.1 [17:16] the jj indicates me, then the launchpad bug number and I like to use a .X to indicate how far into the bisect [17:16] you can use what ever make sense to you [17:17] the important part is the ~ which allows kernels with higher version numbers to install over the bisect kernel without breaking things [17:18] if you are going to upload the kernel to a ppa you will also want to update the release info [17:18] ie. natty-proposed in this example [17:19] if you are using the current dev kernel it will say UNRELEASED and you need to specify a release pocket [17:19] ie. natty, maverick, ... [17:19] however ppa builds are slow and I just about never use them, at least not for regular bug testing [17:20] now you can build the kernel [17:20] fakeroot debian/rules clean [17:20] fakeroot debian/rules binary-headers binary-generic [17:21] this will churn through and should build some .debs that can be installed using [17:21] dpkg -i [17:21] now on to doing the actual bisect [17:23] so bisecting is basically a binary search, start with a known good point and bad, cut the commits in half, build a kernel test if its good, rinse lather, and repeat [17:23] git bisect is just a tool to help you do this [17:24] it is smarter than just doing the cut in half, it actually takes merges and other things into account [17:24] the one important thing to note, for these bisects is if you look at the git log etc, you may find yourself in kernel versions outside of your bisect range [17:25] this is because of how merges are handled, don't worry about it, git bisect will handle it for you [17:25] so the basics of git bisect are [17:26] git bisect start [17:26] where bad is the bad kernel and good is the good kernel [17:26] the problem is how do you know which kernel versions to use [17:27] if you are using the upstream kernels for a bisect then the ubuntu version tags are not available to you [17:27] you need to use either a commit sha, or tag in the upstream tree [17:29] sorry lost my internet there for a minute [17:31] if you used the mainline kernel builds to find a good and bad point then you can just use the kernel tags for those, ie. v2.6.36 [17:31] if you used and ubuntu kernel then you can find a mapping of kernel versions here [17:32] http://kernel.ubuntu.com/~kernel-ppa/info/kernel-version-map.html [17:32] alright so I was asked what the debian.master directory is about [17:32] in the Ubuntu kernel we have two directories to handle the debian packaging [17:32] debian and debian.master/ [17:33] this allows abstracting out parts of the packaging [17:34] when you setup a build, the parts of debian.master/ are copied into debian/ and that is used [17:35] the difference between the two isn't terribly important for most people, think of debian as the working directory for the packaging, and master as the reference [17:36] when I had you edit debian.master/changelog above I could have changed things around and had you edit debian/changelog [17:36] however [17:36] fakeroot debian/rules clean [17:36] will endup copying debian.master/changelog into debian/ [17:37] thus if you change debian/change log you have to do a full edit on it every time you do a clean [17:37] so if you are editing debian/ you do [17:37] fdr clean [17:37] edit debian/changelog [17:38] which is the reversion of doing it to debian.master/changelog [17:38] edit debian.master/changelog [17:38] fdr clean [17:38] for me editing debian.master/changelog keeps me from making a mistake and building a kernel with out my edits to the kernel version [17:40] hopefully that is enough info on the debian/ and debian.master/ for now [17:40] and we will jump back to bisecting for a little longer [17:41] so assuming you have your kernel version for good and bad you start, your bisection [17:41] git will put you on a commit roughly in the middle [17:41] then you can do [17:41] fdr clean [17:42] fakeroot debian/rules binary-headers binary-generic [17:42] sorry caught myself using fdr [17:42] this will build your kernel you can install and test [17:43] and then you can input your info into git bisect [17:43] ie. [17:43] git bisect good [17:43] or [17:43] git bisect bad [17:43] the import thing to remember is to not, commit any of your changes to git [17:44] I tend to edit the debian.master/changelog and update the .# at the end of my version string every iteration of the bisection [17:44] you don't have to do this [17:45] you can get away with just rebuilding, straight or if you want doing a partial build [17:46] the partial build is a nice trick if you don't have a monster build machine but it doesn't save you anything early on in the bisect, when git is jumping lots of commits and lots of files are getting updated [17:46] the trick to doing a partial build in the Ubuntu build system is removing the stamp file [17:47] when the kernel is built there are some stamp files generated and placed in [17:47] debian/stamps/ [17:47] there is one for prepare and one for the actual build [17:48] if you build a kernel, and the build stamp file is around, starting a new build will just use the build that already exists and package it into a .deb [17:48] you don't want to do this [17:49] so after you have stepped you git bisect (git bisect good/bad) [17:49] you [17:49] rm debian/stamps/stamp-build-generic [17:50] this will cause the build system to try building the kernel again, and make will use its timestamp dependencies to determine what needs to get rebuilt [17:51] if the bisect is only stepping within a driver or subsystem this can save you a log of time on your builds, however if the bisect updates lots of files (moves lots of commits) or updates some common includes, you are going to end up doing a full kernel build [17:52] so now for the other tack, what do you do if you don't want to mess with the .deb build and just want to build the kernel old fashioned way [17:52] well you build as you are familiar with. [17:52] make [17:52] make install [17:52] make modules_install [17:53] then you need to create a ramdisk, and update grub [17:54] sudo update-initramfs -c k [17:54] will create the ram disk you need if you don't want to mess with the kernel version [17:54] sudo update-initramfs -c -k all [17:54] then you can do [17:55] sudo update-grub [17:55] and you are done [17:55] so QUESTION: After rm debian/stamps/stamp-build-generic do you still do a fakeroot debian/rules clean when doing the incremental build? [17:55] the answer is no [17:56] doing a clean will remove all the stamp files, and remove your .o files which will cause a full build to happen [17:57] so with only a couple minutes left I am not going to jump into a new topic but will mention something I neglected about about the Ubuntu debian builds [17:58] our build system has some extra checks for expected abi, configs, and modules [17:58] when building against an upstream kernel you will want to turn these off [17:58] you can do this by setting some variables on the command line [17:58] fakeroot debian/rules binary-headers binary-generic [17:58] becomes [17:59] skipabi=true skipconfig=true skipmodule=true fakeroot debian/rules binary-headers binary-generic [17:59] this can also be used when tuning your own configs etc/ [18:00] I think I will stop there [18:00] thanks for attending, drop by #ubuntu-kernel if you have any questions }}}