PackagingBasics

This page is kept for archive purposes. Check out PackagingGuide for more up-to-date information.

Before we begin

  • There are a lot of useful script and apps that have been developed over the years to make packaging .debs easier. I'll be using several of them in my session. If you want a head start install:

devscripts debhelper dpatch cdbs dh-make patchutils fakeroot lintian pbuilder

Also, it would be a great help if you can create an edgy pbuilder before class starts:

1. If you are running Edgy skip to step 2. If you are running Dapper download Edgy's debootstrap from packages.ubuntu.com and install it with dpkg -i

2. Install pbuilder 

sudo apt-get install pbuilder

3. Create a pbuilder by running: 

sudo pbuilder create --distribution edgy \
        --othermirror "deb http://archive.ubuntu.com/ubuntu edgy universe multiverse"

This command will create the pbuilder environment in /var/cache/pbuilder/ by default, you can change the path by editing /etc/pbuilderrc

4. Wait. debootstrap is creating a minimal ubuntu chroot environment so it can take some time.

IRC Log

Welcome to the MOTU School. These sessions are presented by the Universe development community (MOTU) as a way of helping Ubuntu users to get involved with Ubuntu development. See http://wiki.ubuntu.com/MOTU for more info. Today what I will be presenting is not in any way a comprehensive reference to packaging. What I want you to get out of this is some understanding of the tools and techniques that people use to make and maintain source packages.

First of all, we need to know what a source package is. A source package is actually made up of 3 different files:

  1. .dsc (description) file that holds md5sums of the other files
  2. .orig.tar.gz which is the source tarball as you would download it from the program's homepage
  3. .diff.gz that holds any changes we make to the .orig.tar.gz

Today we are going to look at a particular source package, plotdrop. It's a fairly easy package, and was my first package from scratch. Ok let's make a temp directory where we can work in: 

mkdir -p ~/motuschool/plotdrop
cd ~/motuschool/plotdrop
apt-get source plotdrop

Note 1: you need to have the source repo's (deb-src) enabled in your /etc/apt/sources.list
Note 2: unlike apt-get install, you should not run apt-get source via sudo

OK, now you have 3 files (the source package) and a directory (which is the unpacked source package). Let's have a look around: 

ls plotdrop-0.5/

and we see that this a small C program with typical files (COPYING, Makefile, Changelog, etc.). In fact, if we were to extract the tarball from the plotdrop homepage the only difference we would find is that debian directory. If you want you can look at the diff to the source tarball by: 

zless plotdrop_0.5-0ubuntu1.diff.gz

Question from ubuntu_demon: The diff is between upstream and ubuntu ? or between debian and ubuntu ?

Answer: Between upstream (what you get from the app's homepage) and Ubuntu or between upstream and Debian if you are packaging for Debian or Ubuntu uses the Debian package without modification.

Now, let's actually dig into that debian/ directory and see what's going on: 

cd plotdrop-0.5/debian/
ls

We see several files and a patches directory. All of the packaging info and patches to the source need to produce a good quality .deb are in here. What we need to do know is go through what the files are for.

Let's start with changelog: 

less changelog

The first thing to note is that the format of the changelog is particular. dch is a program (found in the devscripts packages) that helps you with the formating. One the first line you will find the package name, version, and release. Each time a new source package is uploaded to the Ubuntu build machines it should have a higher version than the previous one. The version goes like this: (program version)-(debian revision)ubuntu(ubuntu revision)

In this case the program's version is 0.5, the Debian revision is 0 because at the time it wasn't in Debian, and the Ubuntu revision is 1 .

Question from FLeiXiuS: It's not automated by DCH right?

Answer: dch just creates a template

Question from redguy: what happens whem a package get included in debian?

Answer: the debian revision goes to 1 as in 0.5-1

Question from neutrinomass: The current kdeadmin source package version is "4:3.5.4-0ubuntu2 " . What does the '4:' mean ?

Answer: That is called an epoch which is used when upstream versioning has changed or package version mistakes. This assures that a newer package is installed even if the versioning scheme has changed. See http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version for more info.

Question from matid: Is it possible to append to a current changelog with dch?

Answer: Yes. Check man dch to find out about the options.

Question from gnube: Are there packages in Ubuntu that aren't in debian?

Answer: Yes. If a package doesn't isn't in Debian then (Debian revision) = 0 . If the package doesn't need any changes (which is what we like) then we just keep the Debian versioning. We only change the version when we need to.

Ok, back to the changelog file. Next we see various notes as to what was changed from one version to the next and finally a "who done it" line that shows who made the changes and at what time. The things to note here are that the name and email address you use are important and will be used to track who did what and if you digitally sign the package you will need to make sure the email address is the same that is in your gpg key but we'll get to that later

Ok, on to the control file 

less control

The control file holds a lot of information about the source package and it's resulting binary package (.deb). It is arranged in field: value form and most of the fields are fairly self-explanitory but the values are, as you will see, more difficult to determine.

For the source package there is:

  • Source: the source package name
  • Section: what section of the repository does the program belong in
  • Priority: is the program essential for a properly running Linux system?
  • Maintainer: who is taking on the responsiblity, mostly for Debian
  • Build-Depends: the packages that must be installed for the source package to be able to build the .deb
  • Standards-Version: which version of the Debian Policy does this package adhere to

There are more fields available but these are found in most all packages. Check out the Debian Policy for more info: http://www.debian.org/doc/debian-policy/ch-source.html and http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-sourcecontrolfiles

Note that Priority is not random, there are specific values you can have and they are explained in the Debian Policy and Ubuntu Packaging Guide

Question from gnube: An improperly formatted control file can cause problems, correct?

Answer: Yes, it will fail to build. The packaging files are often read by various scripts so it is important to watch the formatting.

Question from gnube: Is there an automated tool to determine Build-Depends information?

Answer: Not really. There are a few scripts out there that can help, but mostly you need to figure out what the program needs to build and add them to Bild-Depends. Let's say a program uses GTK, it needs to have libgtk2.0-dev in the Build-Depends in order to compile correctly. Reading the README file in the source tarball as well as the programs homepage are a good idea.

For the binary package there is:

  • Package: binary package name, you can have more than 1 per source package so you need to specify
  • Architecture: what architecture the binary will build on. "any" means it should build on all archs and "all" means it doesn't have any arch dependent files (i.e. a program written in an interpreted language like python)
  • Depends: the packages that the binary package depends on
  • Description: First line holds the short description. Subsequent lines hold the long description. Be careful of the first space in each line, it is important.

This is again pretty self explainitory.

OK, let's move on to compat real quick. The compat file just has an integer that corresponds to the version of debhelper (a package helper that we will talk about later). In this case since we had a Build-Depends: on >= 4.0.0 , compat has 4 .

On to the copyright file. This file is very important to us, as a open source distro, because it holds the copyright and license information for all the files in the source package. The important things to have are:

  1. where you got the source tarball
  2. document who the source tarball copyright holder is (an include a year)
  3. the License under which we are allowed to distribute the code

For common licenses like the GPL you can put the preample and then a notice that has the location of the full copyright in /usr/share/common-licenses . If the license is *not* one of those in /usr/share/common-licenses then you will need to inclued *all* the text of the license. A common mistake is for the packager to just take the word of the program's homepage for the license. The packager need to make sure that the source code has licensing information and that all the files in the source package are accounted for. People often forget artwork and documentation (which is often under a different license than the code itself)

Question from matid: What if different parts of the program we want to build are covered with different licences?

Answer: You list them all. What files belong to what license and if one of the licenses isn't in the "common" list you have to include the full text. Also check if they're compatible when used together (example: GPL and OpenSSL licences)

It's a bit tedious and seemingly unimportant but it is vital to Ubuntu that we are legally distributing code. There are thousands of packages in Debian/Ubuntu already and many times, people ask "why isn't XYZ in Ubuntu?" and often times it is because of licensing problems.

Question from ubuntu_demon: Is there some recommended place to go for licensing questions ?

Answer: If you are uncertain about a license, do a search on the debian-legal mailing list archives and also check out the Debian Free Software Guidelines. Ubuntu doesn't have a legal team per se, we mostly rely on Debian Legal and our excellent archive admins.

It's good to encourage software authors to use a good (DFSG free) license. Sometimes just saying "Man, this software rocks, I'd like to get it into Ubuntu, but it can't because of the license" is enough to get people to relicense.

Next lets tackle the real meat of the source package, the rules file. 

less rules

The rules file is a Makefile that is run by dpkg-buildpackage to build the .deb . I'm not going to go into a lot of detail here because we don't have time, but notice the various rules (clean, build, install, etc.) and also notice that we aren't installing relative to / but instead to $(CURDIR)/debian/plotdrop/ . This can cause a lot of problems for people at the beginning. All of the dh_* are scripts that are a part of the debhelper build helper (remember we had a dependecy on that in control). Each one has a specific task (usually you can get an idea from the name) and they have good manpages so it is a good idea to man each one to see what it does. One thing you will learn about packaging is that there is usually more than one way to do a task. Debian developers like to make scripts and automate things so for debian/rules :

  1. we could not use and helper system at all and manually run all the commands like an ordinary Makefile

  2. we could use debhelper (as in this case) to provide automation of specific tasks (dh_install installs files)

  3. we could use CDBS (common debian build system), which is an even more automatic system, and the resulting debian/rules file would only be a few lines long

It is generally a good thing to not use 1) because 2) and 3) work well, in most cases, but it is important to know what the automation in 2) and 3) is doing.

Question from ubuntu_demon: using CDBS is easier than debhelper?

Answer: It can be, but not in every case. Some packagers avoid CDBS because it seem like a magical black box, but if you go through and really learn what it's doing, then it can greatly simplify debian/rules. I've literally seen some debian/rules files that were 1 line. I would recommend learning debhelper first before using CDBS as CDBS is more-or-less an automated debhelper.

Question from matid: In what case it might be necessary to modify debian/rules by hand after running debhelper?

Answer: Well, debhelper is just a collection of thos dh_* scripts. So for instance, dh_strip strips the debugging symbols from the binaries, dh_installman installs man pages, and dh_python adds to the postinst script items for byte compiling the code on the user's machine. That's why it's important to look at the man page of each one you are using to make sure you actually need it and that you are using it correctly.

Question from matid: Ok, but dh_make creates rules file, doesn't it?

Answer: Yes, a template rules file. dh_make is used to make a template debian/ dir. Much of the stuff you don't need, particularlly the .ex files. There is a significant amout of scripting help (dh_make, dch, debhelper, etc.) but in the end, you are responsible for making sure what is in debian/ is correct. dh_make can give a good guess and give you a template to fill out but don't assume it put in all the correct answers for you Wink ;-) Packaging is fairly complicated but it can be learned with a little effort and a little help from the MOTUs. Once you get used to it it really isn't bad. There is just a lot of material to keep in mind. This is what makes Debian and Debian-based distros like Ubuntu so solid. I can't just tell you a recipe for writing debian/rules. It's done by knowing what it takes to build the program you are packaging. For the most part, if you program "plays nice" then the dh_make template is pretty close to the final rules file, but if you have to do a lot of tweaking to get it to build and compliy with Debian Policy then you might end up modifying it quite a bit.

The thing to keep in mind is what each element of debian/rules is doing. For instance, do you need dh_strip if you have no binaries?

Question from oly: Can you build packages for programs that do not require building?

Answer: Certainly, we even package docs and artwork with no code at all. Just leave the build: rule empty since there is nothing to build.

Question from redguy: What if upstream Makefile doesn't understand --prefix and installs compiled binaries into some hard-coded directory. Should the packager modify the upstream Makefile or use rules install target to install to debian/package_name?

Answer: You can patch the Makefile. Ideally you could also contact the author to see if they can ship a Makefile that does understand --prefix.

Ok, so we've seen most of the core of the packaging. If we look around we see plotdrop.1 which is a man page. I had to make the man page because the author didn't provide one. Even though this is a GUI app, anything that goes in /usr/bin should have a man page. We see plotdrop.manpages, which is a file that tells dh_installman what man page to install. plotdrop.menu is a Debian menu file. Debian has it's own menu system (which Ubuntu doesn't use) but as I wanted this package to also go into Debian I added it. dirs is a file that has the directories that need to be created when running rules that aren't created by the program's Makefile

Question from matid: What about Ubuntu? Will plotdrop create a menu entry in Ubuntu?

Answer: Yes, it installs a .desktop file which is the freedesktop.org standard for menu files which Gnome, KDE, and XFCE adhere to.

Whenever you do work for Ubuntu think about whether the work is Ubuntu specific or not. If it is not make sure to push that work on upstream (either to Debian if it is packaging, or the authors if it is bug fix code, etc.)

Question from matid: So it's possible to create a package that installs a menu entry in both debian and ubuntu, whichever it's installed in?

Answer: It will actually go into both the Debian menu and the Gnome/KDE/XFCE/etc. menu

Let's look at the patches directory, 01_Makefile.dpatch is a patch that I made to the programs Makefile because it's PREFIX was hardocoded and it didn't want to install to <source dir>/debian/plotdrop/ . What I did is rather than just modifying the Makefile and having it show up as a diff in the .diff.gz I used a patch system called dpatch. It is called in the debian/rules file and patches at build time, that way I can seperate patches by purpose. So if upstream "fixes" the problem that I need the patch for I can easily get rid of it. Which has actually happened in this case, so in the next version I can drop that patch easily.

Question from matid: Is it ok to use dpatch ever for, let's say - typos in the package's description?

Answer: You don't need to use patches for changing files in debian/ as that is part of the packaging. The purpose of the patch systems is to seperate what we have done from the author's code so someone can clearly and easily see what we have changed.

If you want to learn more about patch systems I'll refer you to pitti's excellent MOTU School session: MOTU/School/PatchingSources.

OK, now let's have some fun: 

cd debian/
dch -i

This open up changelog in your favorite text editor (you can set EDITOR in your .bashrc or use update-alternatives) with a new template changelog entry. The thing to keep in mind is to make sure to check that the versioning is correct (it won't add the ubuntu versioning if it didn't exist already) and make sure to check the release (dapper, edgy, etc.) to make sure it is the one you are targetting the source package for.

Question from ubuntu_demon: how do you change the name and e-mail address for the template ?

Answer: You can set the DEBFULLNAME and DEBEMAIL environment variables, in .bashrc for instance. Also make sure that they are the same as the info in your GPG key.

My new changelog entry looks like this: 

plotdrop (0.5-0ubuntu2) edgy; urgency=low

   * Blah blah blah

  -- Jordan Mantha <mantha@ubuntu.com>  Thu, 10 Aug 2006 18:54:14 -0700

OK, so once you have saved changelog go up to the source dir: 

cd ..

Now we are going to build a new source package since we made a change Wink ;-) 

debuild -S

or if you don't have a gpg key run: 

debuild -S -us -uc

and if you have a gpg key but debuild -S doesn't work use: 

debuild -S -k<yourkeyid>

What this does is:

  • runs the clean: rule in debian/rules

  • creates a new diff.gz with the changes you've made

  • runs lintian, which is a helper app that finds common mistakes

  • and creates the new .dsc and signs it if you have a gpg key

Now you can go up a dir again 

cd ..

and now you have a new source package:

  • plotdrop_0.5-0ubuntu2.dsc
  • plotdrop_0.5-0ubuntu2.diff.gz
  • plotdrop_0.5.orig.tar.gz (which hasn't changed)

The plotdrop_0.5-0ubuntu2_source.build file has the source package build log and the plotdrop_0.5-0ubuntu2_source.changes file is a file that is used to upload the source package (it contains the md5sums of all the package files)

One thing that is useful is creating a debdiff: 

debdiff plotdrop_0.5-0ubuntu1.dsc plotdrop_0.5-0ubuntu2.dsc \
 > plotdrop_0.5-0ubuntu2.debdiff

A debdiff is what a MOTU would like to see attached to a bug report. It's much smaller than uploading the whole package somewhere, is only one file, and it allows you to see small changes easily. In this case we just have the changelog entry

Now we probably want to build a .deb to make sure everything went ok and files go where they are supposed to and that installation works, etc. This is where pbuilder comes in. You can build the .deb using debuild but you aren't sure if your dependecies are right and you might be running dapper and want to build a .deb for edgy. pbuilder uses debootstrap to create a clean chroot environment which is a very minimal Ubuntu install. In fact it has the same packages that the Ubuntu build machines have so if a package builds with pbuilder it should also build in the Ubuntu build machines.

Instructions for setting up a pbuilder are at the top of the page, in the Packaging Guide (help.ubuntu.com) and on the Wiki (wiki.ubuntu.com/PbuilderHowto)

If you have a pbuilder created run: 

sudo pbuilder build plotdrop_0.5-0ubuntu2.dsc

and it will unpack the pbuilder chroot and build the .deb inside of that. When it's done it will clean the environment and spit out the .deb to /var/cache/pbuilder/result/

Question from matid: Is it possible to build 2 packages at the time?

Answer: Yeah, it'll be slow, but it will unpack the pbuilder environment to a different temp directory.

Question from oly: will the package also work in dapper even though we are building it with edgy in mind ?

Answer: Not necessarily. You would want to create a dapper pbuilder for that. For instance, on my machine I have breezy, dapper, edgy, Debian unstable, and Debian stable pbuilders. If the dependecies haven't changed then you can get away with it.

Question from ubuntu_demon: Creating a backport (edgy to dapper) is just a matter of changing the dependencies and replacing dapper for edgy in the changelog right ?

Answer: In fact, the simplest backport would be to just build the edgy source package in a dapper pbuilder but sometimes the deps change name or version and then you have to change debian/control.

Question from redguy: Do the packages downloaded by pbuilder get cached somewhere?

Answer: Yes. Pbuilder often has to download a lot of packages. It keeps an apt cache outside the pbuilder to keep the packages around for later use. However, the pbuilder is unpacks the base.tgz (compress chroot) each time you build with it so it stays clean.

Question from redguy: why do I have to install the edgy deboostrap in order to make the edgy pbuilder?

Answer: Because each release has a deboostrap script that tells it what packages to get so the edgy debootstrap has scripts for all previous Debian and Ubuntu releases. But since the devs didn't know in Dapper what Edgy would be they couldn't add an edgy script into dapper's debootstrap package.

You can also create chroot environments to test in using dchroot. I have chroots for dapper, edgy, and Debian unstable on my dapper box. So if I create a .deb for one of those I can test it inside a chroot. DebootstrapChroot is a wiki page that describes setting up a chroot.

For people who don't have pbuilder up yet you can grab the .deb at http://packages.ubuntu.com/edgy/math/plotdrop

What we want to do is check to make sure that the packages go where we want them to, or expect them to at least. To do that you can run dpkg -c on the .deb and it will spit out the list of files that will be installed.

Question from matid: Do we have to manually run lintian -i plotdrop_0.5-0ubuntu2.dsc? It's not automatic, is it?

Answer: Well, it is run during debuild -S but the -i flag gives you more info and you can also run lintian on the .deb to see if it sees a problem with the binary.

Question from redguy: Actually lintian is complaining when I run debuild -S - "changelog-should-mention-nmu". What does nmu stand for?

Answer: NMU is Non-Maintainer Upload and is Debian specific. In Debian each package has a specific maintainer or maintainers and other developers are not supposed to just change another maintainer's package but sometimes it is needed (a release critical bug fix and the maintainer is out of town or something). For a NMU you use a differnent version and put NMU in the changelog, but Ubuntu doesn't have maintainership like Debian. In Ubuntu any MOTU can upload any Universe package, we work as a team.

If you run dpkg -e on the .deb it will extract the control info into ./DEBIAN . In this case it contains the control file, which has all the Dependencies. You can make sure that everything is ok there. Also the postinst (post install) and prerm (pre remove) scripts.

Question from ranf: What's FTBFS?

Answer: Fails To Build From Source, this is what we call source packages that fail to build.

The End

Ok, so I want to encourage everybody to contribute to Ubuntu. Here are some resources:

To start contributing you need to have a GPG key (https://help.ubuntu.com/community/GnuPrivacyGuardHowto if you need help there) and a Launchpad account. Then join the revu-uploaders Launchpad team (https://launchpad.net/people/revu-uploaders) and then upload to REVU (see the REVU wiki page for details) which is at http://revu.tauware.de . You package will be reviewed by MOTUs. If it is a brand new package it will need to get 2 votes to enter Universe.

CategoryArchive

MOTU/School/PackagingBasics (last edited 2008-08-06 16:28:25 by localhost)