Ubuntu Wiki

Ubuntu Open Week - Packaging 101 - Session 1 - Daniel Holbach - Wed, Oct 24, 2007

see also Tuesday session.

19:13 <+dholbach> What I want to show in today's sessions is two tasks Ubuntu Developers do quite often: 1) updating a package to a new upstream release, 2) providing a debdiff.
19:14 <+dholbach> Why are they important? These tasks are performed quite often and if you're able to make good use of them, it's a very good starting point to get involved as an Ubuntu Developer.
19:14 < seb128> dholbach \o/
19:14 <+dholbach> How do you become an Ubuntu Developer? First of all you become a MOTU: a Master Of The Universe. The process is all written up on http://wiki.ubuntu.com/UbuntuDevelopers - basically it boils down to:
19:14 <+dholbach>  1) contribute patches and new packages
19:14 <+dholbach>  2) get them reviewed by the reviewer team
19:14 <+dholbach>  3) get them uploaded by a sponsor
19:15 <+dholbach>  4) Go back to 1) a couple of times until your sponsors tell you "I'm tired of this - you should be able to do this yourself."
19:15 <+dholbach>  5) Write your application to the MOTU Council.
19:15 <+dholbach> http://wiki.ubuntu.com/SponsorshipProcess explains how uploads get sponsored. Sponsoring means: somebody who's allowed to upload to the build daemons, will sign your package with their GPG key and upload it for you.
19:15 <+dholbach> The review process is also quite easy: you file a bug with the a link to the source package or attach the patch (debdiff), then you subscribe either ubuntu-main-sponsors (for packages in main or restricted) or ubuntu-universe-sponsors (for packages that live in universe or multiverse).
19:16 <+dholbach> Are there any questions about that?
19:16 < amarillion> Can you use your own email address and gpg key for new packages before you pass them to a sponsor?
19:16 <+dholbach> Any questions about the processes? How to become an Ubuntu Developer? What MOTUs do?
19:16 <+dholbach> amarillion: yes, good question
19:17 <+dholbach> amarillion: you're encouraged to have your own name in the debian/changelog entry, etc
19:17 <+dholbach> that way they will show up on your http://launchpad.net/~<LaunchpadID>/+packages page
19:17 <+dholbach> which is important when you apply to become a MOTU
19:17 <+dholbach> it will show what you've done over the release cycles
19:18 <+dholbach> the only thing the sponsor does is: read the diff, make sure it's OK, then sign it with their key and upload it
19:18 <+dholbach> Any more questions?
19:18 <+dholbach> Seems everybody in here knows about MOTU and Ubuntu Development already. :-)
19:18 <+dholbach> (or has been in yesterday's session... :-))
19:19 <+dholbach> To sum it up: http://wiki.ubuntu.com/SponsorshipProcess and http://wiki.ubuntu.com/UbuntuDevelopers are the process pages that you really want to bookmark, if you're interested in packaging and joining the team. :-)
19:19 <+dholbach> The MOTU team can be found on #ubuntu-motu and at http://wiki.ubuntu.com/MOTU - if you need help with anything let us know, there's also our motu-mentors list at http://lists.ubuntu.com/ubuntu-motu-mentors
19:19 <+dholbach> If there are no more questions, let's move on and try to update a package
19:20 <+dholbach> the example I'll use is a slightly historic one, we'll try to update brasero 0.5.2 to brasero 0.5.90
19:20 <+dholbach> first of all: make sure you have a   deb-src  line in your /etc/apt/sources.list
19:21 <+dholbach> I have this one for example:
19:21 <+dholbach> deb-src http://archive.ubuntu.com/ubuntu gutsy main restricted universe multiverse
19:21 <+dholbach> that way we don't only have a source for binary packages (which get installed during updates, etc), but also a source for source packages
19:21 <+dholbach> after that, run:
19:21 <+dholbach>  sudo apt-get update
19:22 <+dholbach> next thing will be to set the DEBFULLNAME and DEBEMAIL variables
19:22 <+dholbach> edit ~/.bashrc in your favourite editor and add something like
19:22 <+dholbach> export DEBFULLNAME='Daniel Holbach'
19:22 <+dholbach> export DEBEMAIL='daniel.holbach@ubuntu.com'
19:22 <+dholbach> after that, run       source ~/.bashrc      or restart your terminal
19:23 <+dholbach> these variables will be used by a variety of packaging tools and make your life easier
19:23 <+dholbach> <mzungu>_ QUESTION: There is a desktop tool, already packaged as a .deb - but not in ubuntu universe etc.  how does one do the work to get it included?
19:23 <+dholbach> mzungu_: the process for that is quite easy: it's explained at http://wiki.ubuntu.com/UbuntuDevelopment#NewPackages
19:24 <+dholbach> you will have to upload the source package (not the .deb), which includes .orig.tar.gz .diff.gz and .dsc
19:24 <+dholbach> it will be reviewed, then uploaded if it's OK
19:24 <+dholbach> now we'll install some necessary tools:
19:24 <+dholbach>  sudo apt-get install devscripts build-essential wget cdbs fakeroot liburi-perl debhelper
19:25 <+dholbach> we'll base the new version of brasero on the old one (0.5.2), get it by running:
19:25 <+dholbach>  dget -x http://people.ubuntu.com/~dholbach/motu/brasero_0.5.2-0ubuntu1.dsc
19:25 <+dholbach> normally you'd just use    apt-get source <package>        but in our case, we use an older version
19:26 <+dholbach> if you look at what it downloaded, you'll notice, that there's a .orig.tar.gz a .diff.gz and a .dsc file
19:26 <+dholbach>  * .orig.tar.gz is the upstream tarball we get from the project's homepage
19:27 <+dholbach>  * .diff.gz is the compressed set of changes we apply in Ubuntu to make it build with our build services
19:27 <+dholbach>  * .dsc is a file which includes descriptions as text-only
19:27 <+dholbach> dget -x has downloaded and extracted the tarball, then applied the changes
19:27 <+dholbach> now we'll get the new version we want to upgrade to
19:28 <+dholbach>  wget http://people.ubuntu.com/~dholbach/motu/brasero-0.5.90.tar.gz
19:28 <+dholbach> we'll extract it:
19:28 <+dholbach>  tar xfz brasero-0.5.90.tar.gz
19:28 <+dholbach> and rename it to conform with the naming pattern:
19:28 <+dholbach>  mv brasero-0.5.90.tar.gz brasero_0.5.90.orig.tar.gz
19:29 <+dholbach> in the case of brasero the only packaging changes are in debian/ so in this example, we'll just copy them over
19:29 <+dholbach>  cp -r brasero-0.5.2/debian brasero-0.5.90/
19:29 <+dholbach> <hendrixski> QUESTION: is it possible for a company to host a proprietary service by hosting their own password-protected repository?  Assuming it's not open-source-software that they're hiding behind the passwords?
19:29 <+dholbach> hendrixski: yes, they can do that, I believe sftp is an option
19:30 <+dholbach> best to check the apt manpages
19:30 <+dholbach> now that we've copied over the debian changes, we'll do update the version number and add a changelog entry
19:30 <+dholbach>  cd brasero-0.5.90
19:30 <+dholbach>  dch -i
19:30 <+dholbach> every change we make in Ubuntu needs to have a changelog entry
19:31 <+dholbach> as we're updating to 0.5.90 we'll change the version number in the first line
19:31 <+dholbach> to 0.5.90-0ubuntu1
19:31 <+dholbach> "0.5.90" is the upstream version
19:32 <+dholbach> "-0" is the bit that refers to the debian version (in our case we did not merge a debian version, so we'll use -0)
19:32 <+dholbach> "ubuntu1" says that it's the first version of it in Ubuntu
19:33 <+dholbach> as upload target (behind the version number) we'll use "hardy" instead of "gutsy"
19:33 <+dholbach> we can always only upload to the active development release, which now is hardy
19:33 <+dholbach> gutsy is 'locked'
19:33 <+dholbach> there's also gutsy-updates and gutsy-security, but that's not part of our example right now :)
19:34 <+dholbach> as version string we'll use something like "New Upstream Release"
19:34 <+dholbach> did that work for everybody up until now? any questions?
19:34 <+dholbach> <mbt> QUESTION:  When building testing packages, say, when testing a patch or something, what sort of version scheme should be used to ensure that the test package is (a) newer than the current package, and (b) "older" than the next-to-be-released?  (Example:  LP 131526 is one such instance)
19:35 <+dholbach> mbt: say you're preparing a 0.5.4-3ubuntu2 upload (which fixes 0.5.4-3ubuntu1)
19:35 <+dholbach> you could use 0.5.4-3ubuntu2~testing1
19:35 <+dholbach> ~ is used in cases like that to indicate "this is less than"
19:36 < mbt> Excellent, thank you.  That one has been bugging me.  :)
19:36 <+dholbach> <hendrixski> hhmm... and.. that leads me to another question I guess:
19:36 <+dholbach>  QUESTION: Is it possible to create a nightly build of the main branch that creates package(s) for testing as well?  Are there already tools that do this or would one have to write their own script?
19:36 <+dholbach> hendrixski: you mean a CVS checkout or something?
19:37 <+dholbach> hendrixski: it should be easy enough to script that and use 'dch' to generate a new automated changelog entry for every checkout
19:37 <+dholbach> I'd expect a few lines shell or python should do that
19:37 <+dholbach> everybody has the changelog done alright?
19:37 < DaveMorris> hendrixski: I have that already, I'll paste bin it for you
19:38 <+dholbach> now we'll install the build-dependencies of brasero, so we can build it properly
19:38 <+dholbach> sudo apt-get build-dep brasero
19:38 < hendrixski> DaveMorris, I would appreciate that, thank you :-)
19:38 <+dholbach> so what are build-dependencies?
19:39 <+dholbach> if you check out debian/control you will notice that in its first stanza it's talking about the source and Build-Depends
19:39 <+dholbach> in the next stanza(s) it mentions Depends
19:39 <+dholbach> Depends are what will be required to install the resulting binary package
19:39 <+dholbach> Build-Depends are the packages that are needed to build the package
19:40 <+dholbach> the Ubuntu build daemons all have a minimal build environment, which will just install what you specify in the Build-Depends
19:41 <+dholbach> to build the source package, please run
19:41 <+dholbach>  debuild -S -sa
19:42 <+dholbach> please ignore the (possible) message about you not being able to sign the source package
19:42 <+dholbach> that's irrelevant for now
19:42 <+dholbach> in general you will need a gpg key, everything related to it, is explained over here: https://help.ubuntu.com/community/GnuPrivacyGuardHowto
19:42 < DaveMorris> hendrixski: and anyone else http://www.pastebin.ca/748368
19:43 <+dholbach> we should have a bunch of files now and it should look something like this:
19:43 <+dholbach> brasero_0.5.2-0ubuntu1.diff.gz  brasero_0.5.2.orig.tar.gz        brasero_0.5.90-0ubuntu1.dsc           brasero_0.5.90-0ubuntu1_source.changes
19:43 <+dholbach> brasero_0.5.2-0ubuntu1.dsc      brasero_0.5.90-0ubuntu1.diff.gz  brasero_0.5.90-0ubuntu1_source.build  brasero_0.5.90.orig.tar.gz
19:43 <+dholbach> everybody got that far?
19:43  * mbt nods
19:43 <+dholbach> neat-o :)
19:43 <+dholbach> let's try to build it
19:43 <+dholbach> please run:
19:43 <+dholbach>  debuild -us -uc
19:43 <+dholbach> this will build the package but not try to sign it in the end
19:44 <+dholbach> <mbt> QUESTION:  Can this build process be adjusted for SMP systems?  e.g., is there a way to pass -jX to 'make' where X is the number of parallel process desired?
19:44 <+dholbach> mbt: that should happen automatically
19:45 <+dholbach> does that work out for all of you? I get an error message
19:45 < hendrixski> DaveMorris, thanks.  :-)
19:45 < mbt> I ask because it just tried to build this package (and failed), but only ever used one of my CPUs
19:45 <+dholbach> can somebody pastebin the result?
19:45 < DaveMorris> http://www.pastebin.ca/748370 - coz I delete the last 1
19:45 <+dholbach> mbt: in my case it uses both cores, you could ask on #ubuntu-devel about that
19:46 <+dholbach> DaveMorris: I was asking about the result of the build
19:46 < mbt> http://pastebin.com/m2a85b9fc
19:46 < mbt> That is as much as my terminal history permits
19:46 <+dholbach> #
19:46 <+dholbach> data-disc.c:107: error: expected specifier-qualifier-list before ‘GtkTooltips’
19:46 <+dholbach> #
19:46 <+dholbach> data-disc.c: In function ‘brasero_data_disc_get_property’:
19:46 <+dholbach> exactly what I get too
19:47 <+dholbach> this is a build failure, nick named FTBFS (fails to build from source)
19:47 <+dholbach> what can we do to try to fix it?
19:47 < mbt> New upstream, I would guess.
19:47 <+dholbach> mbt: good answer
19:47 <+dholbach> MOTU work or work as an Ubuntu developer is always detective work
19:48 <+dholbach> if you come across bugs like this one, you'll like check what upstream did in the meantime
19:48 <+dholbach> or see what's happening in Debian and other distributions
19:48 <+dholbach> if you're interested in a package it's good to be in touch with all of there people
19:48 <+dholbach> and that's a great thing about packaging: you get to know a lot of people
19:48 <+dholbach> let's get the new upstream source, 0.6.1 is out already:
19:48 <+dholbach>  wget http://ftp.gnome.org/pub/gnome/sources/brasero/0.6/brasero-0.6.1.tar.gz
19:49 <+dholbach> <amarillion> QUESTION: debuild will create new files, and the "clean" target might forget to remove some files. Is that a problem? Will they be added to the diff?
19:49 < seb128> dholbach: GtkToolTips being deprecated in the new GTK I would look if they build with some GTK DEPRECATED option ;-)
19:49 <+dholbach> seb128: upstream fixed it :)
19:49 < seb128> alright ;-)
19:49 <+dholbach> amarillion: that's usually a sign of a broken clean target
19:49 <+dholbach> amarillion: best to make sure you remove those autogenerated files in the clean target, in debian that's as far as I know a RC bug
19:50 <+dholbach> <mbt> QUESTION:  If, however, a new upstream is not available, I would assume that the proper thing to do is patch it.  Do you modify *.orig.tar.gz, though, or do you do something else to achieve the desired result?
19:50 <+dholbach> mbt: no, we try to leave the .orig.tar.gz as it is, there are only a few cases where it's OK to change it
19:50 <+dholbach> mbt: there are a lot of different packaging patch systems you can use
19:50 <+dholbach> dpatch or cdbs's simple-patchsys among them
19:50 <+dholbach> http://wiki.ubuntu.com/PackagingGuide explains about all of them
19:51 <+dholbach> patches are then stored in debian/patches and applied at build time
19:51 <+dholbach> that's a really nice way to keep track of patches, because once you update to a new upstream release, you might just drop some patches, because they were committed to upstream SVN by removing the patch file
19:51 <+dholbach> good question though :)
19:52 <+dholbach> let's go through the steps to update to 0.6.1 again
19:52 <+dholbach>  mv brasero-0.6.1.tar.gz brasero_0.6.1.orig.tar.gz
19:52 <+dholbach>  tar xfz brasero_0.6.1.orig.tar.gz
19:53 <+dholbach> cp -r brasero-0.5.90/debian brasero-0.6.1
19:53 <+dholbach>  cd brasero-0.6.1
19:53 <+dholbach>  dch -i
19:54 <+dholbach> again, we'll change the version number to 0.6.1-0ubuntu1
19:54 <+dholbach> and the upload target to hardy
19:54 < DaveMorris> [Question] when does hardy as a target open up?
19:54 <+dholbach> <FayZee> Quote's dholbach: "in debian that's as far as I know a RC bug". What is RC?
19:54 < eolo999> whois mdomsch
19:54 <+dholbach> RC means release-critical
19:55 <+dholbach> DaveMorris: it's open already
19:55 <+dholbach> now we'll build the source package again
19:55 <+dholbach>  debuild -S -sa
19:55 <+dholbach> (and ignore the gpg warnings)
19:55 <+dholbach> check that everything went alright, and attempt to build it again
19:55 <+dholbach>  debuild -us -uc
19:56 <+dholbach> <bahadunn> QUESTION: How about using pbuilder so as not to clutter up things with build-deps etc...?
19:56 <+dholbach> bahadunn: good question, I figured that for this tutorial it might take too long for some (on slow connections) to set up the pbuilder
19:57 <+dholbach> pbuilder will create a minimal environment (just like the build daemons) - a chroot, and install the necessary build-depends in there
19:57 <+dholbach> that way you don't need to install them on your 'main system'
19:57 <+dholbach> http://wiki.ubuntu.com/PbuilderHowto has more information on that
19:57 <+dholbach> <mbt> QUESTION:  If a person is going to manage a mediocre number of packages, how would they go about setting up their own dedicated build server?  PPA is nice, but the storage space is rather small for a large number of packages, particularly if you package just emacs and OpenOffice for two releases.
19:58 <+dholbach> mbt: writing a small script that uses pbuilder should be fine, I guess
19:58 <+dholbach> I've been told that setting up 'dak' requires much more effort
19:58 <+dholbach> <bahadunn> QUESTION: Maybe a seperate pbuilder session sometime then?
19:58 <+dholbach> bahadunn: I believe that PbuilderHowto on the wiki should be sufficient, it's quite good
19:58 < bahadunn> yes I agree
19:58 < sistpoty> pbuilder is not session dependant, you can start is as often as you want (or as your box can deal with it)
19:59 < bahadunn> I asked the question just before you posted the link...
19:59 <+dholbach> ok, great bahadunn, thanks for the question
19:59 <+dholbach> does the package build alright for everybody?
19:59 < bahadunn> no problem
19:59 < amarillion> yes it works
19:59  * mbt nods
19:59 < peppych> Finished running lintian.
19:59 < peppych> think so :)
19:59 < amarillion> I get a complaint that hardy is not a valid ubuntu version
19:59 < pascalin> yes
20:00 <+dholbach> rock on!
20:00 <+dholbach> amarillion: I answered in -chat
20:00 <+dholbach> so what we can do now is compare the old and the new versions of the .deb packages
20:00 <+dholbach> get the old .deb file:
20:00 <+dholbach>  wget http://people.ubuntu.com/~dholbach/motu/brasero_0.5.2-0ubuntu1_i386.deb
20:01 <+dholbach> and run:
20:01 <+dholbach>  debdiff brasero_0.5.2-0ubuntu1_i386.deb brasero_0.6.1-0ubuntu1_i386.deb      (if you're on i386)
20:01 <+dholbach> <mbt> QUESTION:  Will debdiff work across architectures?
20:01 <+dholbach> mbt: yes, I ran it on amd64 :)
20:02 <+dholbach> debdiff will compare the contents of the two packages and let you see easily what's missing from either of them
20:02 <+dholbach> also it will show you changed dependencies, etc
20:02 <+dholbach> as a maintainer it's really important to check that and be aware of missing files, etc
20:03 <+dholbach> congratulations everybody, we updated a package successfully! :-)
20:03 <+dholbach> any more questions? :-)
20:03 <+dholbach> ok, let's have a five minute break until we move on to the next session