Packaging101

Ubuntu Open Week - Packaging 101 - Mon, Nov 27, 2006

06:00   dholbach        Hello everybody! We'll move on to  Packaging 101  now. ...
06:01   dholbach        If you want to become part of the MOTU team, or help out with packaging in general, apart from knowing how to write or debug code, it's good to know the tools the distribution uses.
06:01   dholbach        This session won't make you a packaging expert, but afterwards you will have heard of some things, which will make it easier to find your way around. There'll be a MOTU session on Wednesday, 15.00 UTC and Saturday on 16.00 UTC, which will cover what the MOTU team does in more detail.
06:02   dholbach        Please run the following command, it will take some time to work in the background.
06:02   dholbach            sudo apt-get install pbuilder; sudo pbuilder create
06:03   dholbach        MOTUs are the "Master of the Universe", the team of maintainers that takes care of the package in Universe and Multiverse - that's where you start as a package maintainer. http://wiki.ubuntu.com/MOTU
06:03   dholbach        Packaging - what does that involve?
06:03   dholbach        * adding enough information to the Upstream source, to make it buildable on a minimal system.
06:03   dholbach        * split the installed files up into separate packages depending on the target audience
06:04   dholbach             - example everybody uses libgtk2.0-0, but not everybody has an interest in libgtk2.0-dev
06:05   dholbach        * make the packages work out of the box
06:05   dholbach        * add copyright information, nice description, documentation, etc.
06:05   dholbach        Package maintenance involves far more than that, Jordan Mantha (Laser_away) will give a talk about that later today at 21.00 UTC and Thursday at 20.00 UTC.
06:05   dholbach        First we'll install some tools we'll need for the session:
06:05   dholbach           sudo apt-get install devscripts dpkg-dev build-essential
06:06   dholbach        I also prepared an example package we'll use also:
06:06   dholbach           mkdir 101; cd 101; wget http://daniel.holba.ch/temp/packaging101.tar.gz
06:07   dholbach        I learned most from looking at existing packages and trying to fix them, so let's take a look at the libsexy package. It's the one in feisty, I broke a bit to illustrate a few common problems you'll face in the daily work as a package maintainer or Ubuntu developer.
06:07   dholbach        If you untar the packaging101 tarball, you'll find the following files:
06:07   dholbach          libsexy_0.1.10-1ubuntu1.diff.gz  libsexy_0.1.10-1ubuntu1.dsc  libsexy_0.1.10.orig.tar.gz
06:07   _tx     Some things I write don't work well with ppc or x86 but work well on the x86_64 namely my system.
06:08   dholbach        Together they make the source package. The .orig.tar.gz is called the 'original upstream tarball', the '.diff.gz' file is the changes we need to do to make the package buildable and working correctly in Ubuntu. The .dsc provides meta information like checksums, etc.
06:08   dholbach        To unpack the source package run
06:08   dholbach          dpkg-source -x libsexy_0.1.10-1ubuntu1.dsc
06:08   dholbach        If you change into the directory and have a look into it, you'll find a debian/ directory, which contains the packaging information.
06:09   dholbach        Let's try to build the package now. You can use dpkg-buildpackage, but running debuild is easier mostly. So to run a build, simply type
06:09   dholbach          debuild
06:09   dholbach        in the directory.
06:09   dholbach        The friendly build script might now ask you to install some build dependencies, after you did that, it should be spitting out some interesting messages.
06:10   dholbach        in my case the build is complaining (among other things) about the following:
06:11   dholbach        sexy-tooltip.c: In function 'sexy_tooltip_new_with_label':
06:11   dholbach        sexy-tooltip.c:143: warning: implicit declaration of function 'gtk_label_new'
06:11   dholbach        sexy-tooltip.c:143: warning: assignment makes pointer from integer without a cast
06:11   dholbach        sexy-tooltip.c:144: warning: implicit declaration of function 'gtk_label_set_markup'
06:11   dholbach        sexy-tooltip.c:144: warning: implicit declaration of function 'GTK_LABEL'
06:11   dholbach        It's a problem I added myself. :-)
06:11   dholbach        We will need to patch the source to make it build again.
06:11   dholbach        If you have a look at debian/rules you will see that it mentions /usr/share/cdbs/1/rules/simple-patchsys.mk - after working with packages for a while that it means you can use cdbs-edit-patch to add a patch. Martin Pitt (pitti) will give a separate session on how to patch packages on [patching source packages will be Tuesday 18:00 and Thursday 17:00 UTC].
06:12   dholbach        First we run     fakeroot debian/rules clean      to clean up our existing build again.
06:12   dholbach        next we run       cdbs-edit-patch 01-fix-gtk-breakage
06:13   dholbach        as I said before, pitti will explain what happens then :-)
06:14   dholbach        (Note:  cdbs-edit-patch  needs  cdbs  installed.)
06:14   dholbach        Once we're presented with a subshell, in a directory like /tmp/cdbs-new-patch.R29983/libsexy-0.1.10.new , we edit  libsexy/sexy-tooltip.c  with our editor of choice
06:15   dholbach        and add a missing     #include <gtk/gtk.h>
06:15   dholbach        just below the existing   #include
06:15   dholbach        now save the file, and hit Ctrl-D
06:16   dholbach        ls debian/patches/       should show our shiny new patch
06:16   dholbach        Now we run         debuild       again and see if we fixed the package.
06:17   dholbach        It looks like our attempt succeeded.
06:18   dholbach        ls ../*.deb    will show us the packages we made
06:18   dholbach        I'll answer a bunch of questions after the next step.
06:18   dholbach        As I explained above, part of the packaging work is trying to make sure that a package builds in a minimal environment also. Why?
06:19   dholbach        Because on our build daemons, a minimal system will be created for each package build to make sure it works in every, even a fresh, scenario.
06:19   dholbach        We will now generate a new source package with the changes we made earlier.
06:19   dholbach        fakeroot debian/rules clean
06:20   dholbach        debuild -S -sa
06:20   dholbach        should take us there.
06:20   dholbach        cd ..; sudo pbuilder build libsexy_0.1.10-1ubuntu1.dsc      should give the pbuilder some work and we can take a look at a few questions.

<xeruno> what's cdbs?

  • cdbs is a collection of scripts that make packaging, especially writing debian/rules (a Makefile to automate the package build) a lot easier, by simplifiying common tasks.

<TLE> Will the script disregard temporary editor files when it does the patch ?

  • I'm not sure what you mean with editor files. It's better to get rid of them. They will show up in the .diff.gz and in case of binary files make the build fail.

<DraxNS> is there a short manual on how to make simple packages? like to create package for krusader from source?

  • I will later post some links to help with packaging, but I fear that packaging will always be a bit tough in the beginning.

<amarillion> Will debuild automagically add all patches in the debian/patches directory

  • the rules in debian/rules (namely simple-patchsys.mk) takes care of that

<bettsp> It only succeeds if my name is Daniel Holback

  • I'm sorry, yes you're right. Either disregard the message or run debuild with -k<your keyid>

[bettsp] You forgot to run the S-expression I added in the next line to fix my misspelling of your name :)

  • yes, I should have said "please add" (it works too when it can't sign the package. that's just a warning)

<proppy> you refer to 'our build daemon' is that kind of pbuilderd ? 

06:26   dholbach        Ok, my pbuilder is 'finished' now - we'll get back to questions later again.
06:26   dholbach        funnily enough the build fails with:
06:26   dholbach        checking for PACKAGE... configure: error: Package requirements (pango >= 1.4.0, glib-2.0 >= 2.4.0, gtk+-2.0 >= 2.4.0, libxml-2.0) were not met:
06:26   dholbach        No package 'gtk+-2.0' found
06:26   dholbach        This is because libgtk2.0-dev is not installed in the minimal environment I was talking about, but how does pbuilder know what to install?
06:27   dholbach        if you edit   debian/control   you will find the line    Build-Depends
06:28   dholbach        This line indicates the packages that need to be installed to build the package correctly. We'll just add  libgtk2.0-dev
06:28   dholbach        after that we'll run   debuild -S  again
06:28   dholbach        please note that this time we don't need to run   fakeroot debian/rules clean   why?
06:29   dholbach        because the build happened in that clean environment, not in the source dir - another good point about using pbuilder
06:29   dholbach        let's run      sudo pbuilder build libsexy_0.1.10-1ubuntu1.dsc         again and see if we get it working now

<levander> What all does fakeroot debian/rules clean do?

  • it runs the clean target of the upstream source (so whatever the upstream maintainer thought should be cleaned), plus dh_clean and rules you set up yourself

<TLE> if I edit the source with emacs, emacs creates a temp file, will the script disregard that when it makes the diff for the patch ?

  • I'm not sure which file you're referring to. I'm emacs illiterate - sombody else please answer. :-). Emacs will create a sexy-tooltip.c~ backup file, and that will be included in the diff if you don't remove it likewise will gedit by default. if you mean backup files, yes, the "debhelper" system is smartyt enough to not ignore them, but get rid of them when you "clean" the package.

<daxelrod> How would we know from the error message mentioning 'gtk+-2.0' that the package we need is 'libgtk2.0-dev'? Is there a convention for figuring this out?

  • excellent question. One answer to that is: experience with packaging. But there are also some indicators of where to look: In the configure script it failed, because it tried to look up in /usr/lib/pkgconfig so if the build fails for you in pbuilder, but works if you run on your system, you can easily check that directory and find out, where and why it went wrong apt-cache search <...> or apt-file search <...> can help also. check configure.ac is another solution 

06:35   dholbach        The pbuilder build worked for me right now, but there are still some things that are weird:
06:36   dholbach        If you run    dpkg -c /var/cache/pbuilder/result/libsexy-doc_*.deb  (which displays the contents of the package), you will find it's nearly empty
06:36   dholbach        it merely contains a couple of changelogs and copyright notices - what went wrong?
06:37   dholbach        If we look into the debian/ directory, you will see that it contains a couple of *.install files
06:37   dholbach        these files mention which directories and files are supposed to be installed into which package
06:38   dholbach        debian/libsexy-doc.install is empty
06:39   dholbach        if you run         debuild     (to build the package again) and     dh_install --list-missing   afterwards, it will present you with a list of files that were not installed into any package
06:40   dholbach        once we add    debian/tmp/usr/share/gtk-doc/html   to    debian/libsexy-doc.install   it will contain something
06:40   dholbach        now we can run     debuild binary   to not have to go through the whole procedure again, but only rebuild the packages
06:41   dholbach        You will find that if you run       dpkg -c libsexy-doc*.deb     again, it will contain the documentation
06:42   dholbach        I introduced some more bugs into the package, but given that we only have 19 minutes left, we should probably move on to some more questions.

<proppy> debian/libsexy-doc.install doesn't exist

  • hum, it existed for me - anyway, it was supposed to be an empty file.

<nmsa> install pbuilder is still on, slow connection, will it install a new env. will get all packages from net? is doing validation on retrieved packages ...

  • it's not installing all packages, but only a very minimal set, needed to build packages

<cjwatson> which packages are considered by debootstrap?

  • build-essential will be contained in any case.

<Florob> Are *.install files cdbs specific?

  • no, they're not. They're scanned by dh_install.

<amarillion> which is the preferred format for documentation? Should you provide docs in a certain format if they aren't provided upstream?

  • manpages are a good example of documentation that sometimes does not exist. It's always good to push it back to the upstream developers afterwards again. I don't have a link on manpages handy, but I could find you one. I have a template I always use to get started on them.

<rmjb> is the *.install file method the correct way for one source package to have multiple binary targets? like a lib, lib-dev, -doc and so on?

  • some people use *.dirs files or run dh_install to debian/<binary package name> on their own, I personally prefer the .install file method

<xhaker> how about policy on packages. I read executables on /usr/bin must have manpages.

  • manpages are not a strict must have, but they come in very handy and users are gratefule for manpages

<somerville32> Is it possible to have pbuilder setup to work in multiple releases concurrently?

  • there's a page on the MOTU wiki about that - if nobody finds it out in the next minutes, I can try after the talk

<xhaker> can i build for any version deboostrap supports?

[sol] Should all packages install to /usr? What's the use of /usr/local?

  • /usr/local is for software you install yourself (like if you use ./configure && make && sudo make install)

<xhaker> pbuilder extracts the minimal system and then downloads additional dependencies.. does it cache them? for how long? where?

  • yes, it does in /var/cache/pbuilder/aptcache/. I think it stores the packages until they're superseded by a newer version 

06:52   dholbach        If you run into any other packaging related trouble and don't know where to look in the documentation, your first check could be similar packages, then ask the people on #ubuntu-motu or on ubuntu-motu@lists.ubuntu.com

[proppy] can i chroot inside a pbuilder environment to debug a packaging failure ?

  • yes, you can sudo pbuilder login can help with that

[jonh_wendell] do we have to do a clean before every debuild? even if we change just 1 char in some file?

  • no you don't - you can run debuild binary to benefit from a previous build and just restart from wherever the build failed. debuild cleans for you unless you use the -nc option

MeetingLogs/openweekedgy/Packaging101 (last edited 2008-08-06 16:39:13 by localhost)