Ubuntu Wiki

Ubuntu Open Week - QA: Using piuparts to test your packages - Lars Wirzenius - Mon, Apr 28, 2008

[17:01] <liw> I guess it's time to start
[17:01] <Amaranth> Yep
[17:02] <liw> this session is meant for people who package stuff for ubuntu (or debian or other distros using the .deb format)
[17:02] <liw> piuparts is a tool for testing whether a package can be installed and removed, or installed, upgraded, and removed
[17:02] <liw> traditionally, that is done by the developer/package maintainer installing it on their own system, and then perhaps removing it, too
[17:03] <liw> piuparts makes that testing more systematic by automating it
[17:03] <liw> this is useful for people making .deb packages: a minimal check of whether anything at all works
[17:03] <liw> case 1: you have a new package, and want to make sure it can be installed and then removed (and purged) to a pristine system
[17:04] <liw> case 2: you have a new version of a package, and want to make sure it can be upgraded from the previous version, already in the archive, and then removed
[17:04] <liw> there are other cases, but mostly for doing distribution wide QA, so we'll ignore those for now
[17:04] <liw> piuparts does not test that the program contained in the package actually works, just that the packaging (postinst and other maintainer scripts) work
[17:05] <liw> for testing whether the program itself works, other tools are needed, such as autopkgtest
[17:05] <liw> piuparts checks for some common errors as well, such as broken symlinks, missing dependencies required by maintainer scripts, and that the package cleans up after itself when removed and purged
[17:06] <liw> this concludes my intro :)
[17:06] <Amaranth> <Mez> QUESTION: If it picks up common errors, why not add these checks to linda/lintian ?
[17:07] <liw> lintian does not actually run any of the package's maintainer scripts, piuparts does; the things piuparts check for are for things that lintian cannot check for because of this
[17:07] <liw> (linda has been removed, all its functionality is now provided by lintian, but the same answer applies)
[17:09] <Amaranth> next?
[17:09] <liw> oh, yeah, sorry: next
[17:09] <Amaranth> <TankEnMate> QUESTION: does piuparts handle dynamically created files? (ie files that were created by post-inst rather than in the package proper)
[17:09] <liw> TankEnMate, yep, piuparts handles everything that postinst does, except it prevents it from starting services
[17:09] <liw> next
[17:10] <Amaranth> <asomething> QUESTION: Will piuparts be run against all uploads to REVU?
[17:10] <liw> piuparts the name is short for "package installation, upgrading, and removal testing suite" (I peeked in -chat and saw someone ask that)
[17:11] <liw> I hope piuparts will be run on all uploads to all parts of Ubuntu; I meant to work on that, but got diverted to other things, so it didn't happen for hardy, but let's see what we can make happen for intrepid
[17:11] <Amaranth> <lucash> QUESTION: is piuparts ubuntu specific or working on debian, too?
[17:12] <liw> lucash, I originally wrote it specifically for Debian; in fact, it needs some tweaking to work really well for Ubuntu, primarily because Debian has all interesting packages in "main", while Ubuntu's "main" is very small
[17:13] <Amaranth> <stdin> QUESTION: do you think piuparts will be added to debuild, as lintian is now, so it can be ran automatically before upload?
[17:13] <liw> stdin, that sounds like an excellent things to suggest to the debuild maintainers
[17:13] <liw> stdin, however, piuparts can be quite slow, and requires root access (for the time being), so it's not a good thing to run it always
[17:14] <Amaranth> Answer to the REVU question is no because piuparts is for binary packages and REVU only deals with source packages. (right?)
[17:14] <Amaranth> <Mez> QUESTION: how often does piuparts produce false positives/negatives - and - with this in mind, what benefits does using it have over doing these tests manually
[17:14] <liw> Amaranth, correct on binary packages and piuparts; I guess it would be possible to test-build the packages in REVU and use piuparts on those
[17:16] <liw> Mez, piuparts is pretty good, but not perfect, and it's just imperfect enough that running it on the entire package archive (20 thousand packages) is a bit of a pain when it comes to sorting the results; for people working on a small set of packags, the problems should usually not be visible
[17:17] <liw> the benefit of testing with piuparts is that it is tireless, and does things the same way every time, which is good for quality control; testing things manually, when it's past 05 AM and you needed to be in bed eight hours ago to catch your 07 AM flight... well, manual testing is not as even in quality as automatic testing is
[17:18] <Amaranth> <lucash> QUESTION: do you recommend to run piuparts in a chroot or virtual machine, as it seems to need root atm?
[17:19] <liw> piuparts builds the chroot itself (I'll get to how piuparts works in a moment); running piuparts in kvm or virtualbox or another virtual machine is probably a good idea, if you're testing other people's packages
[17:19] <Amaranth> <stdin> QUESTION: What was the inspiration for piuparts?
[17:20] <liw> stdin, in 2005 I had a consulting gig for a client to do some QA work on Debian, and I got tired of fixing bugs and testing things manually, so I wrote piuparts instead
[17:20] <liw> next I'll continue with an explanation of how piuparts works, and I*ll take more questions after that
[17:21] <liw> piuparts works by using a chroot; this requires you to run it as root
[17:21] <liw> it installs a minimal debian or ubuntu system in the chroot, using debootstrap, and then installing the package and any dependencies it needs, possibly upgrades to a newer version of the package (but not necessarily its dependencies), and then removes and purges the package
[17:22] <liw> piuparts takes a snapshot of the list of files in the chroot before installing the package, and after removing it, and compares these lists, and complains about any differences
[17:22] <liw> it also keeps track of file modifications
[17:22] <liw> obviously, things that always change, such as /var/log/*, are automatically ignored
[17:22] <liw> to prevent daemons etc from starting, piuparts uses the policy-rc.d mechanism: it creates a /usr/sbin/policy-rc.d that always exits with 101, signalling that no services should start in the chroot; this makes it safer to run piuparts
[17:22] <liw> chroot is not a very safe virtualization mechanism, so if you are testing other people's code, you may want to run it inside kvm or other real virtualization
[17:23] <liw> after you've digested that, questions please
[17:25] <Amaranth> <Amaranth> Is there any sort of override system to make it ignore 'errors' like lintian?
[17:25] <liw> Amaranth, there are some options to disable some tests, but nothing as fancy as lintian has; that's something that should be fixed some day in the future
[17:26] <Amaranth> <gringo> QUESTION: "... takes a snapshot of the list of files in the chroot ..." can it also track changes to existing files?
[17:26] <liw> gringo, yeah, it does that, too
[17:26] <Amaranth> <funderburg> QUESTION: What if you wanted it to check /var/log to ensure a specific log directory got created?  Like /var/log/named for instance?  Can you override it?
[17:27] <liw> funderburg, that's an interesting idea; I hadn't thought of it, so currently the answer is "no, sorry"; also something that needs to be fixed in piuparts in the future: make it easily customizable for different uses
[17:28] <Amaranth> <mbt> QUESTION: Is it relatively easy to implement pipuarts as a hook to pbuilder?
[17:29] <liw> mbt, I haven't used pbuilder hooks, but from what I know, it should be possible; I do things by running pbuilder first, then lintian, then piuparts
[17:29] <liw> (I have a script that does that, actually, but that's irrelevant for this)
[17:30] <liw> if someone has a cool way of integrating pbuilder and piuparts with pbuilder hooks, please file a wishlist bug against piuparts in the Debian bug tracking system (bugs.debian.org), since that's where upstream development happens
[17:30] <Amaranth> Can we get your script? :)
[17:31] <Amaranth> <lucash> QUESTION: is piuparts capable of being used for all Debian supported architectures?
[17:31] <liw> http://liw.iki.fi/liw/unperish/ is my script, but it's still somewhat specific to my particular circumstances, so might not work for others
[17:32] <liw> lucash, as far as I know, yes
[17:33] <liw> so far nobody has asked the really important question: how do you use piuparts
[17:33] <Amaranth> hehe, let's do that one then :)
[17:33] <liw> I thought you'd never ask :)
[17:33] <Amaranth> <Amaranth> QUESTION: How do I use piuparts?
[17:34] <liw> first, you have to have or build a .deb file, which I will refer to as foo.deb in this session
[17:34] <liw> the command line to run piuparts to test foo.deb is.... wait for it...
[17:34] <liw> piuparts foo.deb
[17:34] <liw> this is not the optimal way of doing it, just the simplest way
[17:34] <liw> you may also want to prefix that with "sudo ", so you run it as root, otherwise it won't work
[17:35] <liw> the reason that is not the optimal way is because piuparts installs a debian or ubuntu system into the chroot, and then all the dependencies... and then removes them
[17:35] <liw> that can be a lot of packages
[17:36] <liw> to speed things up a bit, you can "cache" the contents of the chroot (before foo.deb + deps are installed)
[17:36]  * liw waves to Mez
[17:36]  * Mez waves back
[17:37] <liw> the simple way of doing the caching is by having piuparts re-use the tarball that pbuilder creates, the basetgz one
[17:37] <liw> like this: sudo piuparts -p foo.deb
[17:37] <liw> piuparts can also build its own tarball: sudo piuparts -s pink.tar.gz foo.deb
[17:37] <liw> the -s means "save", so you need to run piuparts once with -s
[17:38] <liw> then the remaining time, you can just use it: sudo piuparts -b pink.tar.gz foo.deb
[17:38] <liw> now, the next thing I'm going to tell you is how to interpret the piuparts output
[17:39] <liw> unfortunately, since packages can fail in so innovative and amazing ways, piuparts does not try to analyze things in much detail, it just tells you whether the package test was a PASS or a FAIL, or whether there was an ERROR
[17:39] <liw> but at least it output hundreds of kilobytes of text in which it embeds that result... ;-)
[17:40] <liw> piuparts by default logs everything: all the commands it runs, and all their output, and this is a lot of text
[17:40] <liw> towards the end (not always quite at the end, but usually within the last 50 lines or so), there is a line with PASS, FAIL, or ERROR, and that tells you the result
[17:41] <liw> reading the surrounding log will almost always tell you what went wrong
[17:42] <liw> so it's mostly long, not really difficult as such, but since there's a lot of it, it is daunting, and every time I get to read one, I get a little headache
[17:42] <liw> I've been meaning to fix this since 2005, but the headache has never been big enough
[17:42] <liw> "PASS" means the test succeeded, "FAIL" that it failed, and "ERROR" that some program that piuparts needed to run did not work
[17:43] <liw> ok, question time again
[17:43] <Amaranth> <lucash> QUESTION: what do you expect from piuparts in the future?
[17:43] <liw> lucash, I expect it to be re-thought from the scratch; it's been three years, and we've gained a lot of experience in how it works well and how it does not work well, and also the surrounding world has changed
[17:44] <liw> for example, real virtualization is now a viable option...
[17:44] <liw> exactly how piuparts will change I don't yet know, that discussion needs to happen with the debian and ubuntu developer communities
[17:44] <liw> which I hope to start after my vacation ;-)
[17:44] <Amaranth> <afflux> QUESTION: does the chroot cache need to be upgraded manually?
[17:45] <liw> afflux, yes
[17:45] <liw> pbuilder obviously updates its own, and that's typically good enough for most people, so it's easiest to just use -p
[17:46] <liw> for those of use who need piuparts to build its own chroot... rm -f pink.tar.gz; then run piuparts with -s again
[17:46] <Amaranth> <Amaranth> QUESTION: Will piuparts make me breakfast? ;)
[17:46] <Amaranth> (out of questions)
[17:46] <liw> Amaranth, yes
[17:47] <Amaranth> Awesome, tell it I want scrambled eggs and hashbrowns.
[17:47] <liw> sorry, only the burnt porridge option is currently implemented
[17:47] <Amaranth> Darn, always a catch. :/
[17:47] <Amaranth> Anyhow, I think we're done with questions, you have anything else you want to say?
[17:48] <liw> for those who want to play with piuparts, I could perhaps point out the -a option, which lets you test packages without having a .deb file
[17:48] <liw> piuparts -a hello
[17:48] <liw> that fetches the hello packag e(it's a real package) from the archive and tests that
[17:48] <liw> other than that, I'm done
[17:49] <liw> thank you all, lots of good, pertinent questions! thank you for your interest, and please use piuparts before your next upload :)
[17:49] <Amaranth> Alright, thanks for running this session. I think I'll use piuparts now :)