Ubuntu Wiki

   1 [01:00] <dtchen> all right folks, it being 00:00 utc and such, let's roll on through. i hear there's a release being prepared. :-)
   2 [01:01] <nhandler> :)
   3 [01:01] <dtchen> again, this packaging session will introduce piuparts for package testing, and we'll mention how to use a VM to further test functionality
   4 [01:02] <dtchen> there are some notes (and corrections!) at https://wiki.ubuntu.com/Packaging/Training/Logs/2009-04-23, so please take look and get your prerequisites all shiny
   5 [01:02] <kklimonda> damn, I've managed to stay awake for this session.. any cookies for me? ;}
   6 [01:02] <dtchen> cough, take a* look
   7 [01:02] <dtchen> any questions before we dive in?
   8 [01:03] <kklimonda> Why so late? ;)
   9 === Pollywog is now known as pollywog_
  10 [01:03] <dtchen> (no idea! the schedule rotation deems it so)
  11 [01:03] <dtchen> ok, and with that resounding silence, let's get started.
  12 [01:04] <kklimonda> dtchen: wait - how to configure piuparts?
  13 [01:04] <kklimonda> do i even have to?
  14 [01:04] <eythian> hopefully this VM gets installed before you're too far through.
  15 [01:04] <kklimonda> :)
  16 [01:04] <dtchen> piuparts(8) is a tool for providing a somewhat systematic method for testing whether a binary package can be installed and removed, and/or any of the intermediate upgrade steps
  17 [01:05] <Pollywog> we are installing a vm?
  18 [01:05] <dtchen> kklimonda: as long as you have piuparts installed, that will suffice
  19 [01:05] <eythian> Pollywog: My desktop is intrepid
  20 [01:05] <Pollywog> ic
  21 [01:05] <dtchen> eythian: / Pollywog: you don't necessarily *need* a VM, but it's quite useful for actual usage testing
  22 [01:06] <dtchen> (ok, back to topic)
  23 [01:06] <dtchen> some (many?) people are familiar with building binary packages from source packages on their local systems
  24 [01:07] <dtchen> pbuilder is but one such utility - granted, it's a very nice utility
  25 [01:07] <dtchen> briefly, one would do something akin to: # pbuilder build foo.dsc
  26 [01:08] <dtchen> if everything "goes well", one ends up with ../*.deb (and the other parts of the source package)
  27 [01:08] <dtchen> people often then use dpkg -i (or gdebi or ...) to install ../*.deb
  28 [01:09] <dtchen> however, pbuilder has one fairly glaring omission in the maintenance sense: it doesn't actually test-install/remove/etc. the generated binary packages
  29 [01:09] <kklimonda> dtchen: is it possible to install packages in pbuilder chroot without too much work?
  30 [01:09] <nhandler> Yes kklimonda
  31 [01:09] <dtchen> kklimonda: absolutely, and i'll touch on that briefly, too, with something called pbuilder hooks
  32 [01:10] <dtchen> since kklimonda provided such a nice segue...
  33 [01:10] <dtchen> pbuilder has an extension mechanism known as hooks. take a look in /usr/share/doc/pbuilder/examples/
  34 [01:11] <dtchen> now, piuparts and pbuilder hooks for executing an install/upgrade/removal test are nominally similar
  35 [01:11] <dtchen> where piuparts really differs is the ability to test install/upgrade/removal for an entire archive
  36 [01:11] <dtchen> that's a bit much for pbuilder (although it's certainly possible with some additional scripting)
  37 [01:12] <dtchen> (so, next time you encounter liw online, give him a virtual beer)
  38 [01:12] <dtchen> does anyone have any questions regarding what piuparts is used for?
  39 [01:13] <kklimonda> no
  40 [01:13] <dtchen> ok, now let's look at the functional testing of binary packages
  41 [01:14] <dtchen> piuparts and pbuilder (with hooks) are useful for fairly basic testing, but they don't actually test whether your pbuilt/sbuilt package actually starts, throws pretty screen bling up, crackles/hisses/pops, etc.
  42 [01:14] <dtchen> (err, oops, was that pulseaudio?)
  43 [01:14] <kklimonda> :)
  44 [01:15] <dtchen> to get a sense of whether your package works, you really should use a VM. at the very least, you can install in the pbuilder chroot and invoke the program(s).
  45 [01:15] <dtchen> now that i've set the stage, let's look at the two munged source packages i've placed at http://kernel.ubuntu.com/~dtchen/package-training-2009.04.23/
  46 [01:16] <dtchen> the first is oss-compat, and we'll use that to demonstrate piuparts
  47 [01:16] <nhandler> Should we download the packages?
  48 [01:16] <dtchen> yes, we should have the munged source packages downloaded
  49 [01:17] <kklimonda> done
  50 [01:17] <dtchen> for people who preferred `apt-get source oss-compat', i'll pastebin the debdiff
  51 [01:17] <dtchen> (well, the relevant portion)
  52 [01:18] <nhandler> For everyone else, you can use dget to download the source package
  53 [01:18] <dtchen> http://paste.ubuntu.com/156227/
  54 [01:19] <dtchen> so the basic idea here is that as of jaunty, module-init-tools requires files to have the ".conf" suffix/extension
  55 [01:19] <dtchen> this modification is a fairly straightforward one: we look at the source package and modify debian/postinst
  56 [01:20] <dtchen> (then adjust debian/control, add/modify the changelog entry, etc.)
  57 [01:20] <dtchen> now, when that package is pbuilt using pbuilder [without hooks], the deb will be generated correctly
  58 [01:21] <dtchen> this case is an excellent illustration of where the install pbuilder hook and piuparts shine
  59 [01:21] <dtchen> because the actual error lies in the postinst, pbuilder [without hooks] won't throw any errors
  60 [01:21] <dtchen> so let's actually use piuparts ;-)
  61 [01:22] <dtchen> the syntax is fairly basic:
  62 [01:22] <dtchen> sudo piuparts foo.deb
  63 [01:23] <dtchen> on my system, because i use apt-cacher-ng, the syntax is bit more convoluted:
  64 [01:23] <dtchen> sudo piuparts -m "http://localhost:3142/ubuntu" --skip-minimize -- foo.deb
  65 [01:24] <kklimonda> dtchen: i get " E: Couldn't find package debfoster"
  66 [01:25] <kklimonda> I've added --skip-minimize and it worked
  67 [01:25] <dtchen> if you're using jaunty, you'll probably need the --skip-minimize
  68 [01:25] <dtchen> now, piuparts will create a minimal chroot if you use the above command, but most of you likely have a pbuilder base.tgz (or jaunty.tgz or whatnot) already
  69 [01:26] <dtchen> you can instruct piuparts to use the existing pbuilder base.tgz by passing -p
  70 [01:26] <dtchen> (so, as piuparts runs...)
  71 [01:26] <kklimonda> dtchen: -p ? not -b ?
  72 [01:27] <dtchen> kklimonda: -b is more flexible
  73 [01:27] <kklimonda> i see - it's a shortcut
  74 [01:27] <kklimonda> :)
  75 [01:27] <kklimonda> dtchen: but still when i use my pbuilder base.tgz I get "E: Couldn't find package debfoster" error
  76 [01:28] <dtchen> kklimonda: are you using jaunty's piuparts?
  77 [01:28] <kklimonda> dtchen: yes
  78 [01:28] <kklimonda> 0.35ubuntu1
  79 [01:28] <dtchen> let's troubleshoot that offline
  80 [01:29] <Pollywog> so piuparts is different for different releases?
  81 [01:29] <juanje1> kklimonda: adding --keep-sources-list option probably it works
  82 [01:29] <kklimonda> dtchen: appending --skip-minimize (as adviced in bug #317721) helps
  83 [01:29] <kklimonda> oh, no ubottu here :)
  84 [01:29] <dtchen> Pollywog: ideally, no - however, there is a difference between the Debian and Ubuntu components and priorities
  85 [01:30] <kklimonda> juanje1: yes - it also worked
  86 [01:30] <dtchen> the Ubuntu modification in jaunty helps work around it, but it's imperfect)(
  87 [01:30] <dtchen> (hmm, Home key fail)
  88 [01:31] <asomething> dtchen: -b is used to specify a specific base tarball, right? I've got pbuilder set up for jaunty, intrepid, and sid
  89 [01:31] <dtchen> asomething: correct
  90 [01:31] <dtchen> now, piuparts's output is fairly verbose
  91 [01:32] <dtchen> the parts we're interested in lie toward the end of the output
  92 [01:32] <dtchen> namely, PASS/FAIL/ERROR
  93 [01:33] <dtchen> note the:
  94 [01:33] <dtchen> ERROR: Broken symlinks:
  95 [01:33] <dtchen>   /etc/modprobe.d/oss-compat.conf -> /lib/oss-compat/linu
  96 [01:33] <dtchen> (that's deftly illustrated at http://paste.ubuntu.com/156227/)
  97 [01:34] <dtchen> the process of correcting the error as a maintainer is fairly intuitive
  98 [01:34] <dtchen> we readd the omitted character, regenerate the source package, rebuild it, rerun piuparts on it. (rinse, lather, repeat)
  99 [01:35] <dtchen> any questions thus far?
 100 [01:36] <dtchen> ok, so we've quickly demonstrated how piuparts can assist in catching maintainer script errors
 101 [01:37] <dtchen> now, as maintainers/developers, we're supposed to be testing all our changes
 102 [01:37] <dtchen> due to whatever excuses, many of us don't ;-)
 103 [01:37] <dtchen> functional testing, however, is quite important
 104 [01:38] <dtchen> really, changes must be tested in a VM for both correctness and efficiency
 105 [01:38] <dtchen> the second example munged source package in my kernel.ubuntu.com space is pavucontrol
 106 [01:39] <dtchen> the change that actually makes jaunty's pavucontrol binary package usable is introduced in debian/patches/0002-Handle-noninitialised-data.patch
 107 [01:40] <dtchen> several iterations of that patch were generated before i discovered that upstream's fix wasn't entirely applicable to our source package
 108 [01:40] <dtchen> i.e., sometimes it's not as straightforward as "just applying upstream fixes"
 109 [01:41] <dtchen> briefly, jaunty ships with an older pulseaudio source package with many, many backported alsa fixes
 110 [01:41] <dtchen> these backported fixes differ from the current upstream source package, and therefore there are several assumptions in upstream's pavucontrol fix that aren't applicable to Ubuntu
 111 [01:42] <kklimonda> mhm
 112 [01:42] <kklimonda> so we should use "test before release" as our mantra ? :)
 113 [01:44] <dtchen> for our purposes, however, we're only going to illustrate that using a VM to test changes is the most efficient manner of demonstrating that a bug was fixed
 114 [01:44] <dtchen> (that's bug 359274)
 115 [01:44] <dtchen> if one were to take my munged source package, it is functionally equivalent to jaunty's superceded 0.9.7-1ubuntu1
 116 [01:44] <dtchen> that version crashes as soon as a user attempts to mute System Sounds in the Playback tab
 117 [01:44] <dtchen> thus, testing in a VM (or a chroot or whatnot) is the simplest way to demonstrate the efficacy of a fix
 118 [01:44] <dtchen> absolutely
 119 [01:45] <dtchen> since many people don't have VMs configured, i won't insist that everyone duplicate the crash right this moment, but run through it on your own time, and remember that development really needs this constant testing
 120 [01:46] <dtchen> anecdote: if i had consistently used piuparts on my alsa-* and pulseaudio* source uploads for jaunty, i probably would have eliminated a half-dozen uploads
 121 [01:46] <dtchen> so, what do we take away from this session? using piuparts is good. testing in a vm after using piuparts is good.
 122 [01:47] <dtchen> i'll open it up for general questions now.
 123 [01:47] <Pollywog> are there some Ubuntu-specific tutorials?
 124 [01:47] <Pollywog> for piuparts
 125 [01:48] <dtchen> no, but feel free to pitch in :-)
 126 [01:48] <dtchen> (this session was just an intro to piuparts)
 127 [01:48] <kklimonda> dtchen: i have a minor problem with pbuilder but it's unrelated to this class.
 128 [01:48] <dtchen> kklimonda: right, we can discuss that offline in -motu after this session
 129 [01:48] <kklimonda> ok
 130 [01:49] <dtchen> Pollywog: do you have specific questions on using piuparts in a particular context?
 131 [01:49] <kklimonda> dtchen: thanks for piuparts introduction - i was going to read about it some time ago and then forgot how it's called :)
 132 [01:49] <nhandler> dtchen: Is there a hook for pbuilder to have it run the deb through piuparts?
 133 [01:50] <Pollywog> no, I am really confused because I have not used piuparts
 134 [01:50] <dtchen> nhandler: no, but there's some duplication in /usr/share/doc/pbuilder/examples/B91dpkg-i
 135 [01:50] <Pollywog> I have played with pbuilder a few times
 136 [01:50] <Pollywog> but in Debian
 137 [01:51] <dtchen> Pollywog: ok. very briefly, one uses pbuilder to generate a set of binary package(s) from a source package
 138 [01:51] <dtchen> Pollywog: however, pbuilder by default does not actually test the installation/upgrade/removal of said generated binary package(s)
 139 [01:52] <dtchen> Pollywog: it can be particularly troublesome if errors lie in the maintainer scripts
 140 [01:52] <dtchen> (since we only encounter those at package install/upgrade/removal)
 141 [01:54] <dtchen> Pollywog: try this: sudo piuparts --skip-minimize hello
 142 [01:54] <dtchen> juanje1: i have not tried using jaunty's piuparts on hardy, so i don't know offhand if it's feasible
 143 [01:54] <dtchen> juanje1: one thing to be aware of is the python2.6 transition
 144 [01:55] <dtchen> Pollywog: err, i think i omitted an -a
 145 [01:56] <Pollywog> I will need to get the sources for hello
 146 [01:57] <Pollywog> oh something happened
 147 [01:57] <dtchen> Pollywog: actually, you don't have to retrieve the sources
 148 [01:57] <Pollywog> yes I see that now
 149 [01:57] <dtchen> Pollywog: -a instructs piuparts to retrieve from the repository using apt-get
 150 [01:57] <Pollywog> I added the -a
 151 [01:58] <dtchen> juanje1: i generally filter the output in some other process
 152 [01:58] <dtchen> juanje1: the verboseness of piuparts can be cumbersome
 153 [01:59] <Pollywog> is there a way to adjust the verbosity?
 154 [01:59] <dtchen> Pollywog: not quite
 155 [01:59] <Pollywog> k
 156 [02:00] <dtchen> any other questions before we wrap up?
 157 [02:00] <Pollywog> none here
 158 [02:00] <Pollywog> dtchen: thanks for the class
 159 [02:00] <dtchen> no prob, thanks all!
 160 [02:00] <juanje1> dtchen: yep, thanks :-)
 161 [02:00] <asomething> dtchen: thanks!
 162 [02:01] <dtchen> so, next week is dholbach with "Getting Started with Ubuntu Development"
 163 [02:01] <dtchen> it will take place on 30 April 2009, 06:00 UTC
 164 [02:01] <dtchen> hope to see you here!