2009-06-25
Next Session: TBD |
1 [01:00] <nhandler> Hello everyone. My name is Nathan Handler, and today we will be learning about pbuilder.
2 [01:00] <nhandler> Can I have a quick show of hands from anyone who is here for the session?
3 [01:00] <Quintasan> \o
4 [01:01] <mschafer> Howdy.
5 [01:01] * mbt raises his hand
6 [01:01] <nhandler> If at any point in the session you have a question, feel free to stop me in order to ask it.
7 [01:01] <sdennie> \o
8 [01:01] <nhandler> When packaging, it is very important to test that the package builds prior to uploading it to the official repositories.
9 [01:02] <nhandler> Test building allows us to catch and fix mistakes before the package is made available to thousands of other users.
10 [01:03] <evanrmurphy> sorry I'm late
11 [01:03] <nhandler> The majority of uploads for Ubuntu are for the development release (currently Ubuntu 9.10, Karmic Koala). However, due to the unstable nature of the development release, many developers prefer to stick with a slightly more stable release.
12 [01:03] <nhandler> No problem evanrmurphy ;)
13 [01:03] <nhandler> This is where pbuilder comes into play.
14 [01:04] <nhandler> pbuilder allows developers to create and manage a minimal Ubuntu chroot.
15 [01:04] <nhandler> The great thing about pbuilder is that the chroot can be for any version of Ubuntu, not just the one you are currently running.
16 [01:05] <nhandler> This allows developers to package for the development release while still running the latest stable release on their computers.
17 [01:05] <nhandler> A pbuilder chroot only contains the core essential packages. This allows developers to verify that their build dependencies are correct.
18 [01:06] <nhandler> Once a package is done building in pbuilder, the chroot is left untouched. This means that it can be used to build multiple packages without worrying about it getting cluttered with extra files and applications.
19 [01:06] <nhandler> Right about now, you should be starting to see why so many developers use pbuilder to test their packages prior to uploading.
20 [01:07] <nhandler> For those of you who attended Dustin Kirkland's session about Personal Package Archives (https://wiki.ubuntu.com/Packaging/Training/Logs/2009-05-07), you might be wondering why you would want to use pbuilder instead of simply uploading to your PPA on Launchpad.
21 [01:07] <nhandler> There are a few reasons you might want to do this.
22 [01:08] <nhandler> First, PPAs only support building packages for amd64, i386, and lpia. However, Ubuntu builds packages for several other architecutres. You can use pbuilder to test that your package will build on these archs.
23 [01:09] <nhandler> Since Launchpad PPAs are used by thousands of users, there is a delay between the time you upload your package and the time it begins to build. This delay does not exist with pbuilder (you also do not need to upload the package).
24 [01:09] <nhandler> pbuilder also has several other benefits that will be discussed later.
25 [01:10] <nhandler> Earlier, I mentioned that pbuilder was for creating and managing Ubuntu chroots. That was only partially true.
26 [01:10] <nhandler> pbuilder can also be used to create and manage Debian chroots. This is very useful for users who contribute to both Ubuntu and Debian.
27 [01:11] <nhandler> You can also have multiple chroots for different Ubuntu and Debian versions. I personally have a chroot for intrepid, jaunty, karmic, and sid.
28 [01:11] <nhandler> Finally, for those of you running amd64, you can also use pbuilder to build i386 packages. However, you can't build amd64 packages on i386.
29 [01:12] <nhandler> The ~/.pbuilderrc file is what you will use to configure pbuilder.
30 [01:12] <nhandler> All developers have a slightly different system for handling multiple chroots with pbuilder.
31 [01:13] <nhandler> I keep all of my pbuilder stuff on an external hard drive, and I have a special script to help me switch between my multiple ~/.pbuilderrc files. I have made these files available in a bazaar branch: lp:~nhandler/+junk/pbuilder
32 [01:13] <nhandler> If you decide to use my scripts, be sure to modify the varaible at the top of the files so that it has the correct path.
33 [01:14] <nhandler> You can then do '/media/pbuilder/switchPbuilder karmic' to switch to the karmic pbuilderrc file. Note that this will cause you to lose your current ~/.pbuilderrc file (if it exists). You might want to back it up first.
34 [01:14] <nhandler> The above command assumes that you places the pbuilder files at /media/pbuilder. If they are anywhere else, your command will be slightly different
35 [01:15] <nhandler> If this is the first time using the pbuilderrc file, you will need to do a 'sudo pbuilder create' before you can do anything.
36 [01:15] <nhandler> Any questions?
37 [01:15] <sdennie> Yes, but I'll wait. ;)
38 [01:16] <nhandler> Ok sdennie
39 [01:16] <nhandler> Now we will take a look at pbuilder hooks.
40 [01:16] <Quintasan> I have a suggestion if you allow :P
41 [01:16] <nhandler> Go ahead Quintasan
42 [01:17] <Quintasan> if you are new to pbuilder and need many chroots, try using .pbuilderrc form ubuntu wiki -> https://wiki.ubuntu.com/PbuilderHowto#Multiple pbuilders
43 [01:18] <nhandler> Quintasan: I link to that later. The issue with that is that the setup on the wiki does not work well for other scripts that run pbuilder for you.
44 [01:18] <nhandler> This is because you need to pass a dist option to pbuilder each time with that setup
45 [01:19] <nhandler> Quintasan: However, thank you for linking them to the wiki page (I was going to do that later ;) )
46 [01:19] <nhandler> Now, back to hooks
47 [01:19] <nhandler> Hooks allow us to extend the functionality of pbuilder to make our lives easier.
48 [01:20] <nhandler> I use 3 hooks with pbuilder: B10list-missing (list missing files), C10shell (drops to shell upon FTBFS), and D10aptupdate (run apt-get update before anything).
49 [01:21] <nhandler> I have created a bazaar for these hooks as well: lp:~nhandler/+junk/hooks
50 [01:21] <nhandler> You can place them in /media/pbuilder/pbuilder/hooks. Be sure to leave the prefix on the filenames.
51 [01:22] <nhandler> Hooks beginning with B<digit><digit><name> are executed after the build system finishes building successfully and before copying back the build result.
52 [01:22] <nhandler> C<digit><digit><name> hooks are executed after build failures and before cleanup
53 [01:23] <nhandler> D<digit><digit><name> hooks are executed before unpacking the source inside the chroot.
54 [01:23] <nhandler> There are several other prefixes that you can use for your hooks. You can read more about them in the pbuilder manpage.
55 [01:24] <nhandler> Now, to test out using pbuilder, we are going to try and build the hello package.
56 [01:24] <nhandler> Before doing this, make sure you have 'deb-src http://archive.ubuntu.com/ubuntu/ karmic main' in your /etc/apt/sources.list (This line for other Ubuntu versions should work as well) or the ubuntu-dev-tools package installed.
57 [01:25] <nhandler> You can now do 'apt-get source hello' or 'pull-lp-source hello karmic' to download the hello source package.
58 [01:26] <nhandler> This should consist of a .diff.gz, a .dsc, and a .orig.tar.gz. We will now build the binary package using pbuilder.
59 [01:26] <nhandler> Type 'sudo pbuilder build hello_*.dsc' to start building the binary. You can use 'sudo pbuilder build --logfile build.log hello_*.dsc' if you want to store the build log to a file.
60 [01:27] <nhandler> If you are using my pbuilder files, you should now look in /media/pbuilder/pbuilder/result/karmic, and you should see a nice .deb that you can install (if you want).
61 [01:27] <evanrmurphy> log since 00:01 UTC, for those who just came: http://paste.ubuntu.com/203246/
62 [01:27] <nhandler> You have successfully built the hello package in pbuilder!
63 [01:28] <nhandler> Any questions/problems so far?
64 [01:28] <sdennie> nhandler: Yes
65 [01:28] <nhandler> Go ahead sdennie
66 [01:28] <sdennie> nhandler: You didn't describe how pbuilder was creating the toolchain for the distro you planned to build against or what the technology was to allow that to happen. I'm assuming some fakeroot magic for the latter.
67 [01:30] <nhandler> sdennie: That magic lies in the /usr/lib/pbuilder/pbuilder-createbuildenv file.
68 [01:31] <nhandler> That file uses debootstrap to create the chroot
69 [01:31] <sdennie> Excellent. That's answers the first part.
70 [01:32] <sdennie> Or maybe both parts
71 [01:33] <sdennie> Anyway, don't let me stop you. We can chat offline if I have more questions. :)
72 [01:33] <nhandler> sdennie: debootstrap is the technology that allows it to happen. If you want to get more detailed, feel free to take a look at the debootstrap source, but that is a little more advanced than we will go tonight ;)
73 [01:34] <sdennie> Fair enough. Proceed. ;)
74 [01:34] <nhandler> If you are not using the D10aptupdate script. be sure to run 'sudo pbuilder update' prior to building a package. This will make sure that your chroot has access to the latest packages in the repositories.
75 [01:34] <mschafer> nhandler: So pbuilder-createbuildenv installs both the toolchain and the other packages marked essential?
76 [01:35] <nhandler> mschafer: Yes, it will take care of creating a chroot with all of the packages you need for a very minimal Ubuntu/Debian installation which resembles the build daemons
77 [01:35] <mschafer> Okay, thanks.
78 [01:36] <nhandler> Now, I want to show you all a few more uses for pbuilder.
79 [01:36] <nhandler> There are several instances where having a clean environment to run a script would be useful.
80 [01:37] <nhandler> You can login to a pbuilder chroot using 'sudo pbuilder login'. Once you exit (Ctrl+d), the chroot will go back to the way it was, just like when building a package.
81 [01:37] <nhandler> If for some reason you want to save your changes to the chroot (i.e. If you update the /etc/apt/sources.list file), you can pass the --save-after-login option to the pbuilder command.
82 [01:38] <nhandler> A few weeks ago, Daniel T Chen showed us how to use piuparts to test our packages (https://wiki.ubuntu.com/Packaging/Training/Logs/2009-04-23). By using the --basetgz or --pbuilder flags, you can have piuparts use your pbuilder chroot for testing packages.
83 [01:38] <nhandler> Any more questions?
84 [01:38] <evanrmurphy> nhandler: Yes.
85 [01:39] <nhandler> Go ahead evanrmurphy
86 [01:39] <evanrmurphy> nhandler: You made a comment about pbuilder-dist earlier. Do you recommend not using it?
87 [01:40] <nhandler> evanrmurphy: I didn't mention pbuilder-dist earlier. I was talking about the method described on the wiki page for handling mulitple pbuiler chroots
88 [01:40] <nhandler> I personally do not use pbuilder-dist, but I know a lot of developers who love it.
89 [01:40] <evanrmurphy> nhandler: I got an error on the example you showed until I realized I had to replace "pbuilder-dist <version>" with "pbuilder", so I can see it complicates examples. ;)
90 [01:41] <Ampelbein> nhandler: yes. there are cases where pbuilder won't fail to build, yet the buildd's will. I was given the advice to add pkgbinarymangler as extra build-dep to resemble more closely the buildds. are there any more tweaks like that?
91 [01:42] <nhandler> Ampelbein: Yes, you can use 'sudo pbuilder create --debootstrapopts --variant=buildd' when creating the chroot
92 [01:42] <nhandler> For those of you who are interested in pbuilder-dist, it is part of the ubuntu-dev-tools pacakge
93 [01:42] <nhandler> * package
94 [01:43] <nhandler> I would suggest reading the pbuilder-dist manpage to learn about it, as its usage is slightly different than pbuilder
95 [01:43] <Ampelbein> ok, thanks.
96 [01:43] <evanrmurphy> nhandler: thanks
97 [01:43] <nhandler> If you are interested in learning more about pbuilder, I would suggest reading:
98 [01:44] <nhandler> The pbuilder manpage: http://manpages.ubuntu.com/manpages/karmic/en/man8/pbuilder.8.html
99 [01:44] <mschafer> nhandler: I had been using a fresh install in a virtual machine to do builds in the past. Sounds like pbuilder is better because it gives the minimal set of other packages. Would you recommend pbuilder over a VM? Any other reasons for or against?
100 [01:44] <nhandler> mschafer: I would suggest using pbuilder over a VM
101 [01:45] <nhandler> One reason is that it is more like the buildd's than a VM
102 [01:45] <nhandler> You also have the ability to use pbuilder hooks with it, which really are useful for debugging FTBFS issues
103 [01:45] <mschafer> Can you use pbuilder on an Ubuntu system to run a Debian chroot?
104 [01:46] <nhandler> mschafer: Yes. As I said earlier, I have both Ubuntu chroots and a Debian sid chroot
105 [01:47] <nhandler> mschafer: A pbuilder chroot is also generally faster and easier to use than a VM
106 [01:47] <mschafer> Though I would imagine that I would still need a Debian VM to actually run the application, since I'm pretty sure that Gnome is not an essential package and therefore not in the pbuilder chroot.
107 [01:47] <nhandler> mschafer: pbuilder will *not* help you test that the applications actually runs. It just tests that it builds
108 [01:48] <mschafer> Ah, gotcha.
109 [01:48] <nhandler> And it will pull in any Build-Depends that it needs in order to build the package
110 [01:48] <sdennie> That's a good point. Does the application build outside the chroot and install/deb inside the chroot?
111 [01:48] <nhandler> sdennie: It doesn't install the binary .deb file that it builds. However, the file is placed outside of the chroot so that you can access it later
112 [01:49] <sdennie> nhandler: Right. That's what I mean. But, does the building happen in the chroot or does it download every dep you may need every time within the clean chroot?
113 [01:49] <ajmitch> all of the compilation happens in the chroot (which is why you can setup ccache in the pbuilder config)
114 [01:49] <nhandler> If you want, you can login to a pbuilder chroot and modify the /etc/apt/sources.list file to use your PPA in order to satisfy Build-Depends as well
115 [01:49] <nhandler> Let me give you guys a few more links that might help
116 [01:49] <sdennie> ajmitch: Thanks. :)
117 [01:50] <nhandler> The Kubuntu Ninja Build Environment wiki: https://wiki.kubuntu.org/Kubuntu/Ninjas/BuildEnvironment
118 [01:50] <ajmitch> sdennie: a clean chroot is used for each build, but a cache of previously downloaded build dependencies is kept
119 [01:50] <sdennie> ajmitch: Very cool. Thanks.
120 [01:50] <nhandler> The last link talks about creating an environment that takes advantage of multiple chroots and hooks.
121 [01:51] <nhandler> All of the hooks I mentioned today can be found in lp:~kubuntu-members/pbuilder/pbuilder-hooks
122 [01:52] <nhandler> Any last comments or questions?
123 [01:52] <sdennie> Interesting stuff nhandler. Thanks a lot for the presentation.
124 [01:53] <nhandler> You are welcome sdennie. Logs of this session will be available at https://wiki.ubuntu.com/Packaging/Training/Logs/2009-06-25
125 [01:53] <mschafer> Thank you, nhandler.
126 [01:54] <nhandler> Since we have a few minutes, does anyone have any comments/suggestions about the series of Packaging Training Sessions we have been holding?
127 [01:54] <nhandler> Any topics that you would like to see covered?
128 [01:54] <Quintasan> Thanks for session nhandler, I'm not so new to pbuilder but I learned lot's of intresting things (super ultra top secret ninja hooks :>)
129 [01:55] <mschafer> I noticed there was a recent session on packaging Java apps, but I haven't read the logs. A session on Python apps would be good for me.
130 [01:55] <Quintasan> I'm intrested in packaging core KDE things
131 [01:55] <nhandler> mschafer: That is something that we should probably have. We do have a session on packaging Perl apps coming up as well ;)
132 [01:55] <nhandler> Quintasan: There is a Kubuntu Tutorial Day next week you might like Quintasan
133 [01:56] <nhandler> Quintasan: You also might be interested in JontheEchidna's packaging training session: https://wiki.ubuntu.com/Packaging/Training/Logs/2009-06-18
134 [01:56] <nhandler> Any other comments/suggestions?
135 [01:57] <evanrmurphy> nhandler: Thanks. I'm still wading through the tutorials and trying to get oriented in the Ubuntu dev world.
136 [01:58] <mschafer> I'm currently working with a package that depends on other packages, but the dependencies are heavily patched. Perhaps this is just something that I should learn from an experienced packager, but if it's meaty enough for a training session, that would be good.
137 [01:58] <mschafer> Maybe a session for gnarly dependencies or something like that.
138 [01:59] <nhandler> mschafer: I'm not sure I understand what your suggestion is.
139 [01:59] * Quintasan scans thorugh the LP branches he got, lots of reading
140 [02:00] <evanrmurphy> nhandler: Are there sessions for other aspects of developer work or is it focused on packaging? Or does packaging just about cover it?
141 [02:00] <nhandler> evanrmurphy: Is there a certain aspect in particular you are interested in?
142 [02:00] <mschafer> Probably because I'm too clueless. Let's see. How does one package a software project when upstream has altered other software that are already in packages but don't have the upstream alterations? As I said, this may be something that isn't of general interest. But it's something I'm currently working through.
143 [02:01] <nhandler> evanrmurphy: Pretty much, any topic related to Ubuntu development is fair game
144 [02:01] <nhandler> mschafer: Is there a reason for not updating the versions of the dependencies currently in the Ubuntu archives?
145 [02:02] <mschafer> The dependency upstream doesn't have those changes. The upstream of the package that I'm working on described their changes as so extensive, they're just shy of a fork.
146 [02:03] <evanrmurphy> nhandler: Perhaps a panorama of how everything in the developer community fits together, a more zoomed-out overview. Then again, there's a good bit of that in the documentation.
147 [02:03] <mschafer> Well, this may not be a widespread problem. It's just my current challenge, so I thought I'd mention it. Don't want to take up any more of your night (morning).
148 [02:04] <nhandler> mschafer: Yeah, that is a little too specific in my opinion for a session, but feel free to ask questions in #ubuntu-motu about it
149 [02:04] <mschafer> nhandler: Thanks! And thank you for your time tonight. Very informative.
150 [02:04] <nhandler> evanrmurphy: I think several people have touched on that in their sessions, but having a session dedicated to that wouldn't hurt
151 [02:05] <evanrmurphy> thanks again for the good session
152 [02:05] <nhandler> You are welcome
153 [02:05] <nhandler> Are there any more questions?
154 [02:06] <nhandler> Ok then, thanks for coming everyone
155 [02:07] <nhandler> Remember, you can always find details about the Packaging Training Sessions at https://wiki.ubuntu.com/Packaging/Training
Packaging/Training/Logs/2009-06-25 (last edited 2009-06-25 03:52:56 by cpe-72-229-211-120)