Ubuntu Wiki

Dev Week -- Packaging 101 - Daniel Holbach -- Mon, Sep 1

(12:00:53 PM) dholbach: Some people might know me already, I'm Daniel Holbach, living in Berlin, Germany and always had a lot of love for the MOTU team, and work for Canonical for ~3 years now.
(12:00:55 PM) pedro_: wooohoooo!
(12:01:10 PM) dholbach: I'll try to answer a few of the regular Developer Week questions beforehand
(12:01:24 PM) Myrtti: wooooo
(12:01:37 PM) dholbach: the schedule is up here: https://wiki.ubuntu.com/UbuntuDeveloperWeek - and please use 'date -u' (in the terminal) to find out which UTC time we have right now :)
(12:02:02 PM) dholbach: questions go into #ubuntu-classroom-chat, please prefix them with QUESTION:
(12:02:14 PM) dholbach: I'll take a look over there and copy them all over every once in a while
(12:02:31 PM) dholbach: let's try not to have too many disturbances in the session itself
(12:02:49 PM) dholbach: OK... who's here for the "Packaging 101" session? :-)
(12:02:55 PM) ***dholbach expects a lot of raised hands
(12:02:56 PM) cupe^: I am
(12:02:56 PM) tacone: \o
(12:03:00 PM) sianis: I am
(12:03:00 PM) hckoe_ is now known as hckoe
(12:03:01 PM) jscurtu: ME!
(12:03:02 PM) chienchouchen: i am
(12:03:02 PM) ***techno_freak raises hand
(12:03:04 PM) cupe^: o/
(12:03:04 PM) ***daradib raises hand
(12:03:06 PM) Raeknouhl: here
(12:03:06 PM) Myrtti: _o/
(12:03:08 PM) cupe^: \o/
(12:03:09 PM) ***jrib 
(12:03:09 PM) soulhacker: me me
(12:03:09 PM) ***qense is partially here!
(12:03:10 PM) ***Oli`` raises both hands
(12:03:12 PM) DoruHush: me
(12:03:13 PM) ***Shunpike raises hand
(12:03:13 PM) ***jpds too.
(12:03:18 PM) stefanlsd: here!
(12:03:21 PM) Iulian: Ay
(12:03:22 PM) ed_agemo: me too
(12:03:26 PM) ***kevjava raises hands sheepishly.
(12:03:26 PM) weefred: here!
(12:03:35 PM) didrocks: even though I will not be present o/ :)
(12:03:37 PM) ***xander21c raisees hand
(12:03:41 PM) raseel: Me
(12:03:51 PM) evolvedlight_bet: hello!
(12:04:01 PM) dholbach: excellent... I'm very excited and very happy you're all here
(12:04:20 PM) dholbach: just to make clear what you can expect during the session
(12:04:40 PM) soulhacker left the room (quit: Read error: 104 (Connection reset by peer)).
(12:04:54 PM) dholbach: in this session we're not going to package something from scratch, instead I'll talk you through the bare-bone structure of a package, so the stuff that makes a package build and you will encounter in almost all packages
(12:05:01 PM) dholbach: I'll try to answer as many questions as possible
(12:05:16 PM) dholbach: for more info, I'd recommend https://wiki.ubuntu.com/MOTU/GettingStarted
(12:05:27 PM) dholbach: and of course http://youtube.com/ubuntudevelopers
(12:05:42 PM) dholbach: ok... let's get started
(12:05:53 PM) dholbach: please all install the devscripts package
(12:05:55 PM) dholbach:   sudo apt-get install devscripts
(12:06:26 PM) dholbach: it contains tools we need for packaging and that make your life a lot lot easier
(12:06:49 PM) dholbach: afterwards, please get the source package we're going to look at together today
(12:06:52 PM) dholbach:    dget http://daniel.holba.ch/motu/hello-debhelper_2.2-2.dsc
(12:07:10 PM) dholbach: if I'm too fast or something doesn't work or I'm unclear, please ask in #ubuntu-classroom-chat
(12:07:13 PM) soulhacker [n=nero@210.212.49.15] entered the room.
(12:07:56 PM) dholbach: if you look at the downloaded files you will notice there's a .dsc a .diff.gz and a .orig.tar.gz
(12:08:27 PM) dholbach: it's the first thing you'll notice: we're not talking about .deb files (binary packages), but source packages
(12:09:03 PM) dholbach: when you start doing ubuntu development, you will always deal with those kind of files, so let's take a closer look
(12:09:34 PM) dholbach: the .orig.tar.gz is the original unmodified source code tarball that was released on the homepage of the upstream developers (software authors)
(12:09:52 PM) dholbach: the .diff.gz is the compressed patch we apply to make it build "our way"
(12:09:58 PM) dholbach: what does "our way" mean?
(12:10:27 PM) dholbach: we need to add a bunch of instructional files to be able to apply ONE build process to all kinds of source packages (no matter if they are python, PHP, C, C++, etc)
(12:10:51 PM) dholbach: <Myrtti> dscverify: can't find any Debian keyrings
(12:11:02 PM) dholbach: you can safely ignore that warning
(12:11:18 PM) dholbach: now please run
(12:11:20 PM) dholbach:    dpkg-source -x hello-debhelper_2.2-2.dsc
(12:11:34 PM) dholbach: dpkg-source will then extract the tarball and apply our patch
(12:11:53 PM) dholbach: the .dsc file is used to check the md5sum and so on (it contains a bit of meta-data about the source package)
(12:12:04 PM) dholbach: <raseel> QUESTION : we need to add a bunch of instructional files to be able to apply ONE build process to all kinds of source packages. Could you elaborate this please ?
(12:12:30 PM) dholbach: raseel: some of you might have tinkered with    ./configure; make; sudo make install   when building software from source
(12:12:38 PM) raseel: yes
(12:12:43 PM) dholbach: this applies to most of the packages that use the auto tools (automake, autoconf, etc)
(12:13:13 PM) dholbach: but there's lot of other cases, for example python packages that use distutils need    python setup.py build, python setup.py install  etc
(12:13:36 PM) dholbach: some packages don't have an upstream build system at all, but just contain a few files that need to be shipped
(12:14:19 PM) duluu is now known as mazaalai
(12:14:48 PM) dholbach: I'll get to the "Makefile" part of the source package in a bit, basically we expect a few Makefile targets (clean, install, etc) to be around, so all packages can be built "in the same way"
(12:15:00 PM) dholbach: ok, let's get cracking and dive in
(12:15:13 PM) dholbach: cd hello-debhelper-2.2
(12:15:21 PM) dholbach: and check out debian/changelog
(12:15:50 PM) dholbach: debian/changelog has a very strict format you need to adhere to
(12:16:04 PM) dholbach: fortunately there's the dch tool in devscripts that makes the task easier
(12:17:03 PM) dholbach: each upload specifies: the name of the source package, the revision, the part of Ubuntu (or Debian) it is uploaded too, the changelog entry text itself and who made the particular upload
(12:17:10 PM) dholbach: (plus the timestamp)
(12:17:36 PM) dholbach: whenever you work on packages you need to put some good effort into making it clear WHAT you changed and WHY
(12:18:17 PM) dholbach: in Ubuntu we maintain all packages a one big team, therefore the next uploader should not have to guess where you got a patch from, why it was applied and the tricky conditions under which you made a package build :-)
(12:18:33 PM) dholbach: let's take a look at the topmost entry
(12:18:48 PM) dholbach: the upload has the revision number 2.2-2 and was uploaded to "unstable"
(12:19:15 PM) dholbach: 2.2 (the part in front of the dash) means: this is the upstream release that was packaged
(12:19:43 PM) dholbach: (remember the hello-debhelper_2.2.orig.tar.gz which basically said: these are the unmodified changes that upstream released as 2.2 on their homepage)
(12:19:53 PM) dholbach: <raseel> QUESTION : Does the debian/changelog file exist for only Debian packages ?
(12:19:55 PM) dholbach: good question
(12:20:14 PM) dholbach: no... whenever we make a change in Ubuntu we describe our changes in the same file
(12:20:21 PM) dholbach: what changes is the version number
(12:20:36 PM) dholbach: if I was to change a tiny bit in the package, I'd upload 2.2-2ubuntu1
(12:20:39 PM) dholbach: which would mean:
(12:20:45 PM) dholbach:  - 2.2 was released upstream
(12:20:55 PM) dholbach:  - 2 revisions have been made in Debian
(12:20:58 PM) dholbach:  - 1 in Ubuntu
(12:21:23 PM) dholbach: then I'd forward my change to the Debian maintainer, he'd incorporate it in
(12:21:25 PM) dholbach:  2.2-3
(12:21:36 PM) dholbach: and we could "sync" the package from Debian again
(12:21:46 PM) dholbach: <tacone> QUESTION: so ubuntu counter gets resetted after each debian release ?
(12:22:10 PM) dholbach: tacone: "resetting the counter" would mean: overwriting all Ubuntu changes with the changes that have happened in Debian
(12:22:25 PM) dholbach: if you decide to do that you need to be     V E R Y     careful
(12:22:36 PM) dholbach: and when I say very, I mean
(12:22:40 PM) dholbach:                       
(12:22:40 PM) dholbach: __   _____ _ __ _   _
(12:22:40 PM) dholbach: \ \ / / _ \ '__| | | |
(12:22:40 PM) dholbach:  \ V /  __/ |  | |_| |
(12:22:40 PM) dholbach:   \_/ \___|_|   \__, |
(12:22:41 PM) dholbach:                 |___/
(12:22:52 PM) raseel: :-D
(12:23:07 PM) dholbach: if you're not careful enough you might drop other small bits that were important to Ubuntu users and might be a regression
(12:23:29 PM) dholbach: so in cases where we're not able to sync straight-away (different opinions of maintainers, upstream, etc) we need to merge
(12:23:40 PM) dholbach: <raseel> QUESTION:And if it is NOT a Debian package ?
(12:23:44 PM) dholbach: raseel: can you elaborate?
(12:23:56 PM) dholbach: raseel: you mean if we inherited a package from "some other place"?
(12:24:06 PM) raseel: yes
(12:24:20 PM) dholbach: we'd still add "ubuntu1" to the version string to indicate that we did an Ubuntu-local change
(12:24:55 PM) dholbach: in cases where there are no "ubuntu<n>" revisions the sync happens automatically at the beginning of the release schedule
(12:25:02 PM) dholbach: <tacone> QUESTION: then I don't understand. If debian releases 2.2-3 which number shall my next release (for ubunt) have ?
(12:25:10 PM) dholbach: tacone: if you need to merge, it'd be 2.2-3ubuntu1
(12:25:16 PM) dholbach: (carry over important Ubuntu changes)
(12:25:24 PM) dholbach: <qense> NEW QUESTION: Isn't this a bit weird? If I'm right this means that 2.2-2ubuntu1 could be exactly the same as 2.2-3.
(12:25:34 PM) dholbach: qense: it could, you need to make sure that it IS exactly the same
(12:25:45 PM) dholbach: <jscurtu> QUESTION: What if you take a source from SuSE, would that work?
(12:25:52 PM) dholbach: jscurtu: no, the build process is different there
(12:25:58 PM) dholbach: <techno_freak> QUESTION: what does 0ubuntu1 mean?
(12:26:08 PM) dholbach: it means we introduced a new upstream version in Ubuntu before we got it from Debian
(12:26:16 PM) dholbach: ok... let's move on, just 31m left :)
(12:26:54 PM) dholbach: I hope the concept of debian/changelog is almost clear now - if not, the other sessions this week of the Packaging Guide will enlighten you :)
(12:27:11 PM) dholbach: let's take a look at debian/control together
(12:27:35 PM) dholbach: you will see that it consists of two stanzas (at least two, two now because it's a simple package)
(12:27:50 PM) dholbach: the first one is about the Source package, the following one(s) are about binary package(s)
(12:28:11 PM) dholbach: a source package needs a name, a section and a maintainer
(12:28:37 PM) dholbach: the Standards-Version gives us a clue which version of the Debian Policy (THE document if you need to know about packaging rules) the package complies with
(12:28:52 PM) dholbach: and a very very intersting other bit: Build-Depends
(12:29:12 PM) dholbach: Build-Depends means: this is the bare minimum of packages that are required to build the package
(12:29:29 PM) dholbach: what happens if I upload a revision to the build daemons (soyuz) is:
(12:30:03 PM) dholbach: the will extract the package (just like we did), copy it into a minimal build environment (chroot containing build-essential which gives us make, etc), then install the build-depends
(12:30:07 PM) dholbach: and then attempt to build it
(12:30:36 PM) dholbach: in this case only debhelper is required (of a version >= 5)
(12:30:47 PM) dholbach: let's take a look at the second stanza
(12:31:05 PM) dholbach: it describes the resulting binary packages (all files that are going to be installed into the package go into one package)
(12:31:41 PM) dholbach: it has a package name and description (that turns up in synaptic, adept, etc)
(12:31:54 PM) dholbach: and two interesting bits: Architecture and Depends
(12:32:20 PM) dholbach: Depends is easy to explain: it's the binary packages this resulting binary packages requires to be installed
(12:32:32 PM) dholbach: ${shlibs:Depends} is just a bit incomprehensible
(12:32:50 PM) dholbach: if you run this on a terminal
(12:32:52 PM) dholbach:    apt-cache show hello-debhelper | grep ^Depends
(12:33:01 PM) dholbach: you will get something like this:
(12:33:02 PM) dholbach:   Depends: libc6 (>= 2.5-0ubuntu1)
(12:33:20 PM) dholbach: this means the hello-debhelper package that is in the archive needs libc6 to be installed
(12:33:44 PM) dholbach: the process how we get from ${shlibs:Depends} to libc6 is very interesting and I can only briefly explain it here
(12:34:25 PM) dholbach: basically the build process will figure out which shared libraries the binaries (stuff in /usr/bin/ or /usr/lib/ etc) in our package are linked against and which package they are in
(12:34:39 PM) dholbach: and substitute ${shlibs:Depends} with the right dependencies
(12:35:02 PM) dholbach: this is a very very awesome piece of voodoo that happens automatically during the build :)
(12:35:42 PM) dholbach: "Architecture: any"  means  build this source package individually on all the available architectures in the data center
(12:36:16 PM) dholbach: in the data center we have supported architectures like amd64 and i386 and community ports like powerpc, sparc, hppa, ia64, lpia, etc
(12:36:47 PM) dholbach: if you build C source for example you want the source to be built on each and every architecture individually
(12:37:21 PM) dholbach: if you use "Architecture: all" it will mean: this package can be used on ALL architectures (package that contains artwork or a few python scripts that don't need to be compiled)
(12:38:30 PM) dholbach: <tuxmaniac> QUESTION: Some more explanation on "Section" part of the control file would help
(12:38:37 PM) dholbach: <dholbach> Check out http://www.debian.org/doc/debian-policy/ch-archive.html#s-subsections
(12:38:51 PM) dholbach: <qense> QUESTION: Doesn't ${shlibs:Depends} work in the Build-Depends field? Why not?
(12:38:59 PM) dholbach: no there isn't, you need to specify them manually
(12:39:32 PM) dholbach: there are indicators for build-depends though (configure.in or configure.ac in packages using autoconf, or setup.py in python distutils packages)
(12:39:41 PM) dholbach: <kevjava> QUESTION: I can't find a package "shlibs"...  if it's not a package, what is it?  (or if this is part of the voodoo, where can I read more about the voodoo :))
(12:39:57 PM) dholbach: kevjava: it's just placeholder that will be substituted after the build
(12:40:24 PM) dholbach: it could be #PLEASE_SUBSTITUTE_WITH_SHARED_LIBRARY_DEPENDS# but it's ${shlibs:Depends} :-)
(12:40:42 PM) dholbach: alright, let's crack on
(12:41:00 PM) dholbach: debian/copyright is another critical part of the package
(12:41:38 PM) dholbach: critical for different reasons though - it has little to do with the actual build process, but it makes sure we reflect all the copyright holders, copyrights and upstream authors in the package
(12:42:17 PM) dholbach: there is content we can't ship because of licenses that forbid us to make changes, etc
(12:42:35 PM) dholbach: this section is (when you create a package from scratch) something you need to pay a lot of attention to
(12:42:58 PM) dholbach: Steve Langasek will talk about "How to avoid making Archive Admins unhappy" later this week
(12:43:40 PM) dholbach: basically it need to contain:
(12:43:42 PM) dholbach:  - the upstream authors
(12:43:54 PM) dholbach:  - ALL copyright holders (make sure you check all files)
(12:44:09 PM) dholbach:  - ALL licenses of all files
(12:44:30 PM) dholbach: in most cases it will be easy (COPYING file says GPLv3) and you just need to double-check
(12:44:56 PM) dholbach: be very careful, there's a lot at stake if we slip code into the archive (even worse: on the CDs) we're not allowed to redistribute, etc
(12:45:19 PM) dholbach: once again: the packaging guide has more info about it
(12:45:36 PM) dholbach: https://wiki.ubuntu.com/PackagingGuide
(12:45:51 PM) dholbach: <tacone> QUESTION: so you have to copy the content from all the license files in there ?
(12:46:55 PM) dholbach: tacone: yes, the source tarball should ship the verbatim license texts all itself and you need to reiterate them (and/or point to /usr/share/common-licenses/ if they're available there)
(12:47:12 PM) dholbach: <techno_freak> fnordschrat> QUESTION: i noticed some version strings like "2:1.0~rc2-0ubuntu13"  what does the "2:" mean
(12:47:21 PM) dholbach: sorry, I missed this one earlier
(12:47:25 PM) dholbach: fnordschrat: good question
(12:47:35 PM) dholbach: the "2" is what we call "an epoch"
(12:47:46 PM) dholbach: it allows you to use a lower version number again
(12:48:01 PM) dholbach: a common use-case for this reverting to an older version
(12:48:25 PM) dholbach: so let's say you maintain the package frobnicator in Ubuntu and shipped the safe but boring 2.0.0 version in hardy
(12:48:39 PM) dholbach: 2.0.0-0ubuntu1 for example
(12:48:56 PM) dholbach: in intrepid you decide to update to 2.1.87 because the set of features sounds cool
(12:49:05 PM) dholbach: so it'd be 2.1.87-0ubuntu1 in intrepid
(12:49:23 PM) dholbach: after getting lots and lots of bug reports from users that your software is broken you decide to go back to 2.0.0 again
(12:49:44 PM) dholbach: so you ship    1:2.0.0-0ubuntu1    in intrepid release and everybody would be happy again
(12:49:54 PM) dholbach: <tacone> guess the epoch should be used when upstream changes the versioning scheme either
(12:49:57 PM) dholbach: tacone: exactly
(12:50:02 PM) dholbach: let's try it out
(12:50:42 PM) dholbach: please type:
(12:50:43 PM) dholbach:    dpkg --compare-versions 2.1.87-0ubuntu1 lt 1:2.0.0-0ubuntu1 && echo true
(12:51:07 PM) dholbach: dpkg (which is the ultimate authority when it comes to package versions) tells us that 2.1.87-0ubuntu1 < 1:2.0.0-0ubuntu1
(12:51:20 PM) dholbach: <kevjava> So the epoch is a way to make sure that the version number is always increasing?
(12:51:23 PM) dholbach: kevjava: yes
(12:51:34 PM) dholbach: <techII> QUESTION: I've seen projects that have various utility scripts shiped with them, under licenses other than the main project. How should these be handled?
(12:51:53 PM) dholbach: techII: if we can't allow those scripts to be shipped in Ubuntu, we need to strip them from the original tarball
(12:52:15 PM) dholbach: for example this could be   frobnicator_2.4.6repack.orig.tar.gz    or some such
(12:52:32 PM) dholbach: to indicate it's not the pristine "2.4.6", but a repacked version
(12:52:50 PM) dholbach: Debian uses DFSG as an acronym there - Debian Free Software Guidelines
(12:53:14 PM) dholbach: <daradib> QUESTION: What about using [version]really[version] to revert to an older version, as in 10.0.1.218+10.0.0.525ubuntu1~hardy1+really9.0.124.0ubuntu2 (the version of flashplugin-nonfree in hardy)?
(12:53:47 PM) dholbach: daradib: epochs are a nice feature, they just come with the problem that if we (in Ubuntu) decide to introduce one and the respective Debian maintainer decides to NOT use one, we have a problem
(12:54:06 PM) dholbach: because new Debian revisions will always be smaller than ours, we cannot "sync" any more
(12:54:16 PM) dholbach: let's move on to the last part of the puzzle
(12:54:17 PM) dholbach: debian/rules
(12:54:40 PM) dholbach: the first line of the file already gives it away: it's a Makefile
(12:54:41 PM) dholbach: #!/usr/bin/make -f
(12:55:23 PM) dholbach: those of you who have worked with makefiles already will notice that there are build targets called clean, install, build, binary-indep, binary-arch and so on
(12:56:05 PM) dholbach: you will also notice that in those targets the upstream build system is "wrapped"
(12:56:16 PM) dholbach: ./configure is called, make is called, etc
(12:56:25 PM) dholbach: just with different prefixes and in 'special' places
(12:56:51 PM) dholbach: the dh_* scripts are all part of debhelper (remember, it's the package we build-depended on)
(12:57:36 PM) dholbach: which contains a huge set of very handy helpers to make common tasks like "install this .desktop file and register it in the right place" or "put this changelog in the right place and pretty please compress it" very very easy
(12:58:24 PM) dholbach: it's the piece of the source package that's easy to get wrong, but in most cases the messages during the build are pretty understandable and there are lots of examples
(12:58:44 PM) dholbach: this is the reason why I very much recommend to start working on existing packages, fix small bugs first before moving on to other things :)
(12:58:54 PM) dholbach: please check out https://wiki.ubuntu.com/MOTU/GettingStarted
(12:59:05 PM) dholbach: and the documents linked from there
(12:59:20 PM) dholbach: and please join #ubuntu-motu if you ever have any questions about packaging, etc
(12:59:40 PM) dholbach: we have one minute left, so let's take a break before our next session
(12:59:48 PM) dholbach: Upstream Bug Linkages -- JorgeCastro (jcastro)
(12:59:49 PM) dholbach: woohoo
(12:59:52 PM) dholbach: THANKS EVERYBODY!