KernelMaintenanceStarter

This document is an elaboration over the KernelMaintenance page with a focus on the Debian build system.

Suggested reading

A working knowledge of git, kernel build, debian packaging, chroot etc. will be helpful.

For Git information, see KernelTeam/KernelGitGuide

For SRU related information see StableReleaseUpdates

Please Note

You should always build the kernel on the release that the kernel is targeted for. If you are building a jaunty kernel, please build on a jaunty system or use a chroot environment. As a good general practice always build kernels under a chroot environment.

Building in a chroot environment

This is especially useful when you are building for different architectures that are supported by the CPU (like LPIA (low power Intel architecture) on i386 or i386 on x86_64). Or to build for newer release without having to upgrade the installed distribution.

You can create a chroot environment manually (the hard way) or automatically (the easy way). chroot environment will bindmount the users home directory, deleting files in the environment will delete files in the users home directory as well. If a chroot session is not ended cleanly, it leaves around files under /var/run/schroot//. It may be necessary to unmount all the directories and remove session files manually.

Manual chroot setup

This section is mainly meant for reference. Everything described here is done the same way by the script described in the next chapter. There is an utility called debootstrap that will simplify this task a lot. First install it with apt-get, then call (as root)

Please make sure that you have debootstrap installed on your system:

sudo apt-get install debootstrap

Invoke debootstrap with admin privileges:

sudo debootstrap --arch <arch> <suite> <target>

debootstrap will then create a directory specified under target, for release specified by suite say for example hardy, for architecture specified by arch say for example i386 or LPIA. Note, if you are building kernel for hardy on gutsy please enable gutsy-backports in system->administration->software sources->updates

The following command sets up the chroot build environment.

sudo chroot <target> apt-get install debhelper build-essential fakeroot kernel-wedge

You may want to copy /etc/hosts and /etc/sudoers from the system to the chroot for convenience.

schroot is better than chroot, it allows privileged users to prepare the build environment.

Following is the configuration (/etc/schroot/schroot.conf) setup for schroot in hardy:

[hardy]
type=directory
description=Ubuntu hardy
location=/srv/hardy-chroot
priority=2
users=foo
run-setup-scripts=true
personality=linux32

Note, the personality option is optional, and useful in a amd64 environment.

schroot -chardy

Automated chroot setup

Under the KernelTeam/KernelBuildScripts directory is a git repository containing several helper scripts for building the kernel. You may download them as follows:

git clone git://kernel.ubuntu.com/ubuntu/kteam-tools.git

To automatically create the chroot environment use:

chroot-setup/build-mkschroot [--arch=<arch>] <suite> <target-dir> [<mirror>]

OR

sudo chroot-setup/build-mkschroot --arch=i386 hardy hardy-i386 http://archive.ubuntu.com/ubuntu

Where:

  • arch stands for different arch supported by the CPU, for example i386 on AMD64 or LPIA on i386 etc.

  • suite stands for release name such as hardy, gutsy, intrepid etc.

  • target-dir stands for the directory where the chroot environment will be installed.

  • mirror stands for the http location to download environment from, http://host[:port]/dir/

You don't need to specify a mirror when using the LPIA arch

If you encounter the following error

manjo@adlai:~/devel/ubuntu/kernel$ sudo ~/bin/kteam-tools/chroot-setup/build-mkschroot --arch=i386 jaunty jaunty-chroot  http://archive.ubuntu.com/ubuntu 
debootstrap --arch=i386 jaunty jaunty-chroot http://archive.ubuntu.com/ubuntu
E: No such script: /usr/share/debootstrap/scripts/jaunty
manjo@adlai:~/devel/ubuntu/kernel$ 

copy the latest script (intrepid) to the release name (jaunty) under /usr/share/debootstrap/scripts/

manjo@adlai:~/devel/ubuntu/kernel$ sudo cp  /usr/share/debootstrap/scripts/intrepid /usr/share/debootstrap/scripts/jaunty
manjo@adlai:~/devel/ubuntu/kernel$

The script will create the directory, do the initial debootstrap. Then create a config section for schroot, and pull additional packages that are required for kernel builds.

The naming of the chroots follows the naming scheme on the DC machines, which is <suite>[-<arch>]. The arch part will be omitted if it is the same as the host architecture. For example on i386 the chroot for hardy is named hardy, but on a amd64 it will be named hardy-i386.

Please note that above chroot command will mount your home directory & other system directories under /var/lib/schroot/mount/harty-i386-*/home etc.

/dev/pts on /var/lib/schroot/mount/jaunty-i386-cb08e67a-bea3-465a-9e44-02cf0985f1ba/dev/pts type none (rw,bind)
tmpfs on /var/lib/schroot/mount/jaunty-i386-cb08e67a-bea3-465a-9e44-02cf0985f1ba/dev/shm type tmpfs (rw)
/home on /var/lib/schroot/mount/jaunty-i386-cb08e67a-bea3-465a-9e44-02cf0985f1ba/home type none (rw,bind)

OR

manjo@adlai:~/devel/ubuntu/kernel$ mount | grep jaunty | cut -f 3 -d " "
/var/lib/schroot/mount/jaunty-i386-103b0edb-2972-4ba0-8448-0804bd55da53
/var/lib/schroot/mount/jaunty-i386-103b0edb-2972-4ba0-8448-0804bd55da53/proc
/var/lib/schroot/mount/jaunty-i386-103b0edb-2972-4ba0-8448-0804bd55da53/dev/pts
/var/lib/schroot/mount/jaunty-i386-103b0edb-2972-4ba0-8448-0804bd55da53/dev/shm

So if you remove the home directory under /var/lib/schroot/mount/jaunty*/home, this could do damage to your /home. If you decide to kill the chroot creation process prematurely you may also want to remove these mounts. You could do it as follows:

sudo umount $(mount | grep jaunty | cut -f 3 -d " ")

Basic system setup

Please make sure that the following packages are installed on your system.

  • git-core
  • debhelper
  • build-essential
  • fakeroot
  • kernel-wedge

To install the packages listed above use the following command:

sudo apt-get install git-core debhelper build-essential fakeroot kernel-wedge

Optionally you may also install the following packages, these packages provide the user with build tools that might come handy in building and packaging a new kernel.

  • ccache package: For faster kernel re-build. (Fast compiler cache)

sudo apt-get install ccache

After installing the ccache package, please modify your .bashrc file to include the following. This will ensure that ccache is first in your $PATH environment variable. Please refer man pages for more information on ccache.

if [ -d /usr/lib/ccache ]; then
        export PATH=/usr/lib/ccache:"${PATH}"
        export CCACHE_NLEVELS=5
fi

CAUTION: Setting CCACHE_NLEVELS to a high value causes cache reaping. For instance, setting CCACHE_NLEVELS to 8 caused over 20sec delay between compiles. Good performance has been observed for CCACHE_NLEVELS set to 3.

  • devscripts package: scripts to make the life of a Debian Package maintainer easier.

sudo apt-get install devscripts

Please note that installing devscripts will install/upgrade a large number of dependent packages. please see the file /usr/share/doc/devscripts/README.gz, and for full details on these scripts. The most useful devscript is debuild. debuild can be used to check if build dependencies are satisfied for the current installation.

Optionally you may also install the following packages that might help debuild to function flawlessly.

  • xmlto
  • docbook-utils
  • gs
  • transfig
  • sharutils

sudo apt-get install xmlto docbook-utils gs transfig sharutils

Continue reading without doing a debuild -b if you want to build kernel with debian/rules.

manjo@adlai:~/devel/ubuntu/kernel/ubuntu-jaunty$ debuild -b
 dpkg-buildpackage -rfakeroot -D -us -uc -b
dpkg-buildpackage: set CFLAGS to default value: -g -O2
dpkg-buildpackage: set CPPFLAGS to default value: 
dpkg-buildpackage: set LDFLAGS to default value: -Wl,-Bsymbolic-functions
dpkg-buildpackage: set FFLAGS to default value: -g -O2
dpkg-buildpackage: set CXXFLAGS to default value: -g -O2
dpkg-buildpackage: source package linux
dpkg-buildpackage: source version 2.6.28-7.18
dpkg-buildpackage: source changed by Tim Gardner <tim.gardner@canonical.com>
dpkg-buildpackage: host architecture i386
dpkg-checkbuilddeps: Unmet build dependencies: makedumpfile xmlto docbook-utils
dpkg-buildpackage: warning: Build dependencies/conflicts unsatisfied; aborting.
dpkg-buildpackage: warning: (Use -d flag to override.)
debuild: fatal error at line 1329:
dpkg-buildpackage -rfakeroot -D -us -uc -b failed
manjo@adlai:~/devel/ubuntu/kernel/ubuntu-jaunty$ 

if makedumpfile is missing then make sure you have makedumpfile is installed or setup chroot environment.

Build system breakdown

The kernel is built when executable rules [target] is invoked from linux/debian directory. Commonly used debian target names are as follows.

  • clean
  • binary
  • binary-arch
  • binary-indep
  • binary-generic
  • build
  • build-arch
  • build-indep

Other kernels (each are under a separate source tree) that are supported are:

  • LUM (Linux User Mode) - Limited to Gutsy and Hardy
  • LRM (Linux Restricted Modules) - For all releases.
  • LBM (Linux Backport Modules) - Ubuntu specific, details to follow later in the document.

Building the kernel

To download the kernel source, please follow the documentation in KernelGuide, or use the command below:

Download the latest Linus tree and download the ubuntu-<release> kernel relative to that.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

This will create linux-2.6 directory and download the latest Linus tree there, the advantage is that if you want to download different releases of ubuntu kernel source it will be faster.

git clone --reference linux-2.6/ git://kernel.ubuntu.com/ubuntu/ubuntu-jaunty.git ubuntu-jaunty

If you want to apply any cherry pick patches do the following:

cd ubuntu-jaunty/

git cherry-pick -s -e <submit ID>

You should also change the changelog under debian/changelog

linux (2.6.27-12.28)  UNRELEASED; urgency=low

to

linux (2.6.27-12.28~lpNNNNNNUSERNAME1) UNRELEASED; urgency=low

The resulting kernel will be named

linux-image-2.6.28-7-generic_2.6.28-7.21~lpNNNNNNUSERNAME1_i386.deb

OR you can download the kernel using the following command line:

git clone <user>@zinc.ubuntu.com:/srv/kernel.ubuntu.com/git/ubuntu/ubuntu-<release>.git <dir>

where release stands for hardy/intrepid/jaunty etc.

chroot into the chroot environment you have already created.

cat /etc/schroot/schroot.conf

towards the end of the file you will find...

[intrepid-i386]
description=Ubuntu intrepid (i386)
type=directory
location=/var/chroot/intrepid
users=manjo
run-setup-scripts=true
personality=linux32

to chroot into intrepid-i386 do

schroot -c intrepid-i386

Make sure after you have schroot'ed to your partition, ccache is set up in your path.

NOTE: After you schroot, your /home is now bindmounted to /var/chroot/intrepid/home/ if you delete /home under /var/chroot/intrepid you will damage your /home.

In the top-level kernel directory invoke the fakeroot command. This will build the binary packages for all kernel flavours. The kernel packages will be located (by Debian convention) in the directory one level above the current directory. Other files created during the process are located in the debian directory. In order to compile only a specific flavor, replace arch with your favorite arch, as below.

Now is a good time to check all your dependencies. The debuild command can do this for you. Invocation is:

debuild -d

In karmic you need to do the following because debian/control is now removed from the git tree. The step below will set up the naked git tree for building a kernel/module etc.

fakeroot debian/rules clean

Note: The 'clean' target is a bit of a misnomer here since we moved the generated files (control.stub, control) out of version control. The clean target actually creates these files. It is a known drawback, and unavoidable.

specific arch:

fakeroot debian/rules binary-generic

OR

All archs:

fakeroot debian/rules binary-arch

Cross-compiling for ARM

We cross-compile armel flavours for quick build testing. Steps are outlined here: http://idlethread.blogspot.com/2009/01/recipe-of-day-cross-compiling-armel.html

As schroot creates a file /etc/debian_chroot which contains the name of the chroot, it is easy to create a chroot which automatically does cross compiles. Simply copy the configuration of an i386 chroot and name it with a special suffix (e.g. <series>-armel). Then you can add the following code to your profile (e.g. $HOME/.bashrc):

# THIS MIGHT BE ALREADY PRESENT (start)
# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "$debian_chroot" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi
# THIS MIGHT BE ALREADY PRESENT (end)

if [ "$(echo $debian_chroot|cut -d- -f2)" = "armel" ]; then
        echo "Setting environment for armel cross-compile"
        export CROSS_COMPILE=arm-none-linux-gnueabi-
        export PATH=<your path to arm-buildtools>/bin:$PATH
        export $(dpkg-architecture -aarmel)
fi

Native build ARM package in sbuild and qemu

Cross compiling package for ARM is a quick way for testing, but all our Ubuntu packages are native build even for armel arch. For example, when we upload kernel source package to PPA, the buildd will do native building on a real ARM hardware, which currently is fsl-imx51 board. It will take almost 6 hours to finish the whole building, so it takes too long for developer to know the build result.

With sbuild and qemu, we can do the same native build in our qemu virtual environment. It will slow than cross compiling but it is much faster than native building in a real ARM hardware, moreover it is almost the same as native building.

sbuild: http://manpages.ubuntu.com/manpages/lucid/man1/sbuild.1.html

Install ubuntu-dev-tools on 10.04 Lucid, it might not be available in older release.

sudo apt-get install ubuntu-dev-tools

Setup a lucid armel building environment

mk-sbuild --arch armel lucid

(mk-sbuild is written by Kees Cook, -:))

Build your source package from kernel tree:

kernel-tree-dir$ debuild -S -sa -I.git -I.gitignore -i'\.git.*' -k[your-gpg-key]

Then you will got an *.dsc file with other source package files in the parent directory. Now use sbuild to build your kernel source package:

sbuild -d lucid-armel linux-xxxxxx.dsc

sbuild will start to run qemu and try to setup the native build tools, then eventally start to build the whole source package. Please wait for a while to see whether your building is finished successfully.

Building the Headers

Build and install the binary-headers package before compiling the modules package. This package contains the kernel headers that are required to build the modules. The headers package generated by calling this target is not sufficient to build linux modules. This also requires the flavor specific headers package generated by calling the binary-<flavor> target.

fakeroot debian/rules binary-headers

followed by:

fakeroot debian/rules binary-i386

and

fakeroot debian/rules binary-indep

Building linux-ubuntu-modules

First get the source

  • git clone <user>@zinc.ubuntu.com:/srv/kernel.ubuntu.com/ubuntu/ubuntu-hardy-lum.git <dir>

Then check for the ABI version (debian/changelog) and install the linux-headers (all and flavor specific) package that matches that and the flavor of modules that are to be build.

After that the modules can be build by

  • fakeroot debian/rules binary-modules-<flavor>

You can build a module by itself by doing the following without doing an entire kernel build.

fakeroot debian/rules prepare-<flavor>
fakeroot make -C debian/build/build-generic M=`pwd`/drivers/blah/blah

where flavor is one of generic, rt, ume, xen, etc.

Building kernel udebs

A udeb (or micro-deb) is a special kind of Debian package. The main difference is that a lot of policy requirements are waived. For example, a udeb does not contain a changelog, licence, manpages or md5sums. The reason is to minimize size which is important as the installation takes place completely in RAM, with swap only becoming available after stage 4 of the installation (partitioning).

Kernel udebs are built basically by repackaging a regular kernel image package. Reason is again to reduce memory usage: not all modules included in a kernel image package are needed during an installation. Also, different modules are needed in the initrd for different installation methods, remaining modules can either be loaded later or optionally (by manual selection or through dependencies). The package kernel-wedge contains the toolset used to reorganize a kernel-image package into multiple kernel (module) udebs. (http://d-i.alioth.debian.org/doc/internals/ch03.html)

Sometimes we need to build udebs manually to do some quick kernel building test before we upload kernel source. Currently, we got 2 methods.

  • Use sbuild to build kernel packages and udebs from a kernel source package
  • Manually build udebs:

fdr binary-arch binary-udebs

UKML patch submissions

cherry pick

Do cherry picks with the following options:

  • git cherry-pick -s -x -e SHAid

When it drops you into an editor you need to add the following line with a newline before and after, normally this is placed below the subject line:

  • BugLink: http://www.launchpad.net/bugs/123456

KernelTeam/KernelMaintenanceStarter (last edited 2010-06-02 11:29:44 by cooloney)