2009-07-23

Revision 1 as of 2009-07-23 01:02:09

Clear message

   1 [00:59] <jawnsy> Hi everyone
   2 [00:59] <jawnsy> It looks like it's about time for the Packaging Perl Modules session
   3 [01:00] <jawnsy> Hopefully some of you are here for the talk, or gwolf and I will be talking to ourselves :-)
   4 [01:00] <gwolf> Yup - 2:00AM according to me, so we are ready to start!
   5 [01:00] <jawnsy> If you've got a question, feel free to ask whenever. I only ask that you try to keep questions relevant to the thing currently being discussed, just so the logs don't get too confusing
   6 [01:00] <gwolf> nhandler: So, you are the guy who invited us. Should we expect anything from you?
   7 [01:00] <gwolf> or should we just address The Audience?
   8 [01:00] <maddeth> 1am by me :p but dont know how long I will be here
   9 [01:01] <nhandler> gwolf: You guys are in charge now ;) I'm just here for the show now
  10 [01:01] <gwolf> good :)
  11 [01:01] <jawnsy> Okay, so a quick introduction
  12 [01:01] <gwolf> ok, so lets al just welcome and listen to jawnsy
  13 [01:01] <ajmitch> most people will probably remain quiet :)
  14 [01:01] <gwolf> I have this (annoying?) tendency of having a hard time to sit and listen at an audience, I always end up interrupting
  15 [01:01] <jawnsy> gwolf and myself are members of the Debian pkg-perl team, gwolf is a Debian Developer, and I am not
  16 [01:01] <gwolf> so I'd rather interrupt jawnsy rather than myself ;-)
  17 [01:02] <jawnsy> We're going to do this talk under the assumption of no previous knowledge making any packages, either for Ubuntu or for Debian
  18 [01:02] <jawnsy> Luckily, packaging Perl modules is pretty gentle. A few tools you will need frequently are: dh-make-perl, lintian
  19 [01:03] <gwolf> That's basically thanks to that the Perl modules all share a basic infrastructure
  20 [01:03] <jawnsy> I'm going to go through packaging a small module from CPAN, we'll discuss what commands you can use to do it
  21 [01:03] <jawnsy> first of all, in order to follow along you'll need to apt-get install the aforementioned tools
  22 [01:03] <gwolf> ...And that's also the main reason we (a group of ~15 people at most) can keep up with ~1300 packages in a reliable way!
  23 [01:04] <maddeth> aptitude installing now :)
  24 [01:04] <jawnsy> Another useful package is devscripts, which contains "debuild" -- a command we'll be looking at later
  25 [01:05] <jawnsy> While everyone's installing that stuff, is there a particular module anyone would be interested in using as an example?
  26 [01:05] <gwolf> jawnsy: oh, catalyst could be a good example
  27 [01:05] <jawnsy> Otherwise I'll just pick one at random
  28 [01:05] <gwolf> (of something you want to avoid ;-) )
  29 [01:06] <gwolf> lets start simple, with something not dependency-rich
  30 [01:06] <Ryan52> Locale-Msgfmt
  31 [01:06] <Ryan52> no dependencies other than Perl core, and it should "just work".
  32 [01:07] <jawnsy> okay
  33 [01:07] <jawnsy> So, make a directory and change into it (the build process might clutter your directory with some files, so it's best to put it somewhere). I usually make a bunch of directories like 'tmp'
  34 [01:08] <jawnsy> the first command you want to run is: dh-make-perl --cpan Locale::Msgfmt
  35 [01:08] <gwolf> jawnsy: Let me fill in a minute please :)
  36 [01:08] <jawnsy> Hopefully you have CPAN configured, otherwise it might ask you to configure it. Usually just hitting enter all the way down will be OK
  37 [01:09] <gwolf> I think we have a bit different approaches - I think it's a bit better to first introduce what is _about to_ happen ;-)
  38 [01:09] <gwolf> But anyway - Yes, dh-make-perl is a magic script
  39 [01:09] <gwolf> ... In preparing Debian packages, you will see lots and lots of dh_* calls - To the debhelper scripts
  40 [01:10] <gwolf> dh-make-perl is _NOT_ part of debhelper, but extensively uses i
  41 [01:10] <gwolf> it
  42 [01:10] <gwolf> ...Anyway... Ok, I agree with jawnsy's reasoning (which was sent to me out of band)...
  43 [01:10] <gwolf> ...so I'll let him continue with the demonstration
  44 [01:10] <gwolf> we can later talk about pedagogy ;-)
  45 [01:11] <jawnsy> debhelper, as you may or may not know, is a collection of scripts that Debian uses to build and install packages
  46 [01:11] <jawnsy> prior to debhelper, we were using huge makefiles.. which, as you can imagine, gets pretty hard to maintain and debug
  47 [01:12] <jawnsy> Once you do dh-make-perl --cpan Locale::Msgfmt, you should see a bunch of text scroll by your screen, how it's unpacking a tarball
  48 [01:12] <jawnsy> Most importantly, at the end, you should get a line that says "--- Done" which meant everything went OK
  49 [01:12] <jawnsy> For those following along with this, please confirm that you can build using dh-make-perl. If not, mention any problems you are having in #ubuntu-classroom-chat
  50 [01:13] <jawnsy> Okay. So, it did a bunch of magic. Let's look at what it created.
  51 [01:14] <jawnsy> cd into Locale-Msgfmt-0.14, and list the directory's contents
  52 [01:14] <jawnsy> Everything in that directory is from the tarball we just downloaded from CPAN
  53 [01:15] <jawnsy> what is important to note for us is the files in debian/, which basically tell our installer (debhelper) how to do its magic. It also contains metadata files that tell Aptitude how to resolve dependencies, for example
  54 [01:15] <jawnsy> I'll go through these files (in debian/) file-by-file so we can look at what they are for.
  55 [01:16] <jawnsy> While dh-make-perl is great at its magic, it's also not perfect. So we have to manually check that everything is OK and makes sense.
  56 [01:16] <gwolf> Just as a sidemark - often, dh-make-perl's generated files are good enough for straight _local_ use
  57 [01:17] <gwolf> that means, if you just install them, you can basically trust they will not eat your hard disk, and will even behave
  58 [01:17] <jawnsy> Oh, yes. If you are building Perl modules for your own consumption, they are often good enough to install right away. You can use: dh-make-perl --install --cpan Locale::Msgfmt for that purpose
  59 [01:17] <gwolf> of course, if you are interested in contributing Debian or Ubuntu, a human should check it!
  60 === they is now known as vorian
  61 [01:18] <jawnsy> Often doing this is better than using the CPAN shell to do installs, because this way you can remove packages easily
  62 [01:18] <jawnsy> the debian/compat file just contains the version of debhelper that the module is built for, in this case it should just contain "7" on its own line, which means it's designed for the debhelper 7 series
  63 [01:19] <gwolf> And IIRC, dh-make-perl will not by default work on modules that are marked as unstable by their authors, just on the highest official release (which can be otherwise using the CPAN shell)
  64 [01:19] <maddeth> cat compat
  65 [01:19] <maddeth> lol
  66 [01:19] <jawnsy> yep, it's small :-)
  67 [01:19] <maddeth> oops ;)
  68 [01:19] <gwolf> maddeth: Yup - debhelper uses lots of small files. You will find several such files with minor indications
  69 [01:19] <jawnsy> Now, look inside the 'control' file
  70 [01:20] <jawnsy> I'll briefly discuss some important points, but to get the full idea of what's going on, reading the Debian Policy Manual is the best way
  71 [01:21] <gwolf> right - we got reminded that you might be looking at some possibly different things
  72 [01:21] <gwolf> i.e. if you are running the latest stable versions, or even more if you are using the LTS releases, you will -of course- face older results. Specially LTS will generate a _very_ different set of results (although should be equally usable)
  73 [01:22] <gwolf> but anyway, please ignore such small inconsistencies
  74 [01:23] <jawnsy> Mhm. Basically, this d/control file is what tells aptitude/synaptic various things which you might see as familiar. This is where a package description is defined, among other things
  75 [01:24] <gwolf> debian/control basically lays out all the metainformation - What packages does this one depend on? What is it about? Who created it? ± when? (while it does not contain the dates, as we will go over that soon, it has a rough indication on when it was last updated: its standards-version)
  76 [01:24] <gwolf> ..and several other, less interesting details
  77 [01:25] <gwolf> Note that even if you are working with Ubuntu... at the package source level, keep in mind it will always be debian/*
  78 [01:25] <jawnsy> oh, when I say d/control, I should note that it's a short form (in discussion) for saying debian/control
  79 [01:26] <jawnsy> we get used to that slang a lot in pkg-perl
  80 [01:26] <gwolf> For this module (or package, depending on which side of reality you are standing), the information in debian/control is quite minimal - but were you to build a more involved package, it can... grow quite a bit
  81 === Pici` is now known as Pici
  82 [01:26] <gwolf> heh, in Debian in general, and I guess it also happens in Ubuntu :)
  83 [01:26] <jawnsy> Yes, right now, it only contains two sections (things separated by a newline)
  84 [01:27] <jawnsy> the top part refers to the "source package", the second part refers to binaries generated from the source package. None of that matters right now, so don't worry about it
  85 [01:27] <jawnsy> Okay, onto the next file. You should see a file called: liblocale-msgfmt-perl.docs
  86 [01:27] <jawnsy> and a similar one with the extension .examples
  87 [01:27] <jawnsy> cat these and take a look at them. You'll notice it's just a line-separated list of filenames and paths
  88 [01:28] <gwolf> some of this files might not exist under some circumstances - they depend on what is included on each of the modules you work on
  89 [01:28] <jawnsy> These are magical files, and they trigger debuild to "install" them. Note that anything listed in these files will be installed on Ubuntu and Debian systems which install your package.
  90 [01:28] <jawnsy> Basically, .docs is a way of listing some files which are useful for users to read, so they'll be installed on your system.
  91 [01:29] <gwolf> And, of course, they are autogenerated - so it is possible you will find some examples or files missing. Feel free to just create them with the filenames (relative to the source package's root) for whatever files you want treated as such.
  92 [01:29] <jawnsy> It should list a single file, README, for this package
  93 [01:29] <jawnsy> take a look at the contents of README. you'll notice that it doesn't really say anything interesting.
  94 [01:30] <jawnsy> just the usual stuff like how to install the package manually. this sort of thing isn't tremendously useful to our users, so you can remove the README entry from the .docs file
  95 [01:30] <jawnsy> now, it's no use keeping an empty .docs file around, so since there's nothing we need to install, you may simply rm the .docs file
  96 [01:30] <gwolf> In fact, it could be (and was, several releases ago) simpler: debhelper is quite flexible. here, the filenames are prepended by the package name (liblocale-msgfmt-perl.docs i.e.) - If you were to add other binary packages generated by the same source, they would not be included. you could just name them "docs" and "examples", and the results would end up in all generated files.
  97 [01:30] <gwolf> sorry, all generated binary packages
  98 [01:31] <jawnsy> Is everyone following with us here?
  99 [01:31] <maddeth> could you just remove the *.docs file altogether? or is it a requirment?
 100 [01:31] <jawnsy> You can remove the .docs file if it's empty
 101 [01:31] <gwolf> You can of course remove it if there are no docs to install
 102 [01:31] <jawnsy> And you should.
 103 [01:32] <jawnsy> Okay, the next file to look at is 'rules'
 104 [01:32] <jawnsy> Some of you in the audience might notice that it looks an awful lot like a makefile.
 105 [01:32] <jawnsy> Well, it is :-)
 106 [01:32] <jawnsy> (and is in fact processed by GNU make)
 107 [01:32] <gwolf> Documentation is usually installed to /usr/share/doc/<pkgname>, and examples /usr/share/doc/<pkgname>/examples.
 108 [01:33] <gwolf> jawnsy: In fact, there was a discussion some months (years?) ago in Debian, as to whether debian/rules needed to be a Makefile or could be something else
 109 [01:33] <gwolf> in theory it would work just if it had the proper shebang to anything other than make
 110 [01:33] <gwolf> in the end it was left as a GNU make file fo... consistency
 111 [01:34] <jawnsy> syntactically, makefiles are pretty easy to work with. there's not too much you need to know to work with a makefile.
 112 [01:34] <gwolf> heh, at first sight don't try to make _much_ sense out of debian/rules
 113 [01:34] <jawnsy> depending on the version of dh-make-perl you have installed, you might get files of various length.
 114 [01:34] <gwolf> It basically means: "For every target, call the suitable dh helpers"
 115 [01:34] <gwolf> oh, right!
 116 [01:34] <gwolf> Well, for the latest versions, using the short DH7 variant, it basically says:
 117 [01:34] <jawnsy> Just remember that the rules file you have is what tells debhelper how to build and install your package, and how to clean up after itself
 118 [01:34] <gwolf> #!/usr/bin/make -f
 119 [01:34] <gwolf> %:
 120 [01:34] <gwolf>         dh $@
 121 [01:35] <gwolf> But we used to generate a ~50 line debian/rules until recently.
 122 [01:35] <jawnsy> that's a new shorthand debhelper has, which makes working with and maintaining packages that much easier, especially Perl ones
 123 [01:35] <jawnsy> the last file we haven't looked at yet is watch
 124 [01:36] <jawnsy> you don't need to care that much about that one. It's just a way for Ubuntu and Debian to automatically look for new upstream versions
 125 [01:36] <jawnsy> because everything is on the CPAN, dh-make-perl sets that up for us :-)
 126 [01:36] <gwolf> it basically consists of an URL and a pattern to look for in its webpage
 127 [01:36] <jawnsy> this way, some commands that use it, such as uscan (in package devscripts, which you should've installed by now)
 128 [01:37] <jawnsy> can make use of the information therein. It doesn't really matter for now :-)
 129 [01:37] <jawnsy> okay, so, now that we know what all these control files are doing, change back to Locale-Msgfmt-0.14 (the root of your package)
 130 [01:37] <gwolf> but you can try, just for the sake of it: Run "uscan --verbose"
 131 [01:38] <jawnsy> ah, good point gwolf :-)
 132 [01:38] <gwolf> :)
 133 [01:38] <jawnsy> okay, now, to actually build this package into the binary .deb files we are used to seeing
 134 [01:38] <gwolf> anyway, jawnsy... you were picking up your magic wand
 135 [01:38] <gwolf> please perform the trick
 136 [01:38] <gwolf> note that I have my hands firmly tied behind my back and cannot move.
 137 [01:39] <jawnsy> Make sure you have the 'dpkg-dev' package installed
 138 [01:40] <jawnsy> this is a collection of programs you'll need to build .deb binaries
 139 [01:40] <jawnsy> there are two commands we can use to start the build, either debuild or dpkg-buildpackage
 140 [01:41] <jawnsy> it will display a lot of output. if you've ever installed a CPAN module in the past, you'll note that the output looks mostly the same
 141 [01:41] <jawnsy> this is because the debhelper process is actually running the Perl build system -- that is, running Makefile.PL or Build.PL
 142 [01:42] <jawnsy> Is anyone having problems with the build?
 143 [01:42] <jawnsy> if you do a ls of your current working directory, you'll note that there are some intermediate build files. this is because debhelper built the module and hasn't cleaned up after itself
 144 [01:43] <jawnsy> in order to restore everything back to the "clean" state, run the command: debian/rules clean
 145 [01:43] <jawnsy> or 'debclean'
 146 [01:43] <maddeth> sorry jawnsy where do we run dpkj-buildpackage? the debian directory?
 147 [01:43] <gwolf> you can run it from the base package directory
 148 [01:43] <jawnsy> you want to run it in the root directory of the module, so in this case, Locale-Msgfmt-0.14
 149 [01:43] <maddeth> thankyou
 150 [01:44] <gwolf> (you will notice my hands have been untied and you have some shiny new packages ready!)
 151 [01:44] <jawnsy> the reason for that is that dpkg-buildpackage looks for files in debian/* under the base directory
 152 [01:44] <jawnsy> once you have done the build, change to the directory *above* the base directory (this is why I told you to make a temporary directory at the beginning)
 153 [01:45] <jawnsy> you should notice a bunch of files there. Most importantly to this process is the .deb file, which was magically built for you by dpkg-buildpackage
 154 [01:45] <gwolf> In fact, we could have got to this precise place even in a more automated way to begin with :-) If we had specified the command «dh-make-perl --build --cpan Locale-Msgfmt», the package would have been built and left in our currently working directory. Many times I have done that.
 155 [01:45] <jawnsy> if you so desire, you can install it on your system using: sudo dpkg --install liblocale-msgfmt-perl_0.14-1_all.deb
 156 [01:46] <jawnsy> but, how do we know what it's going to install? how can I be sure it won't overwrite some really important files?
 157 [01:46] <gwolf> Well, to begin with... The dpkg package management system would refuse!
 158 [01:46] <dam> gwolf: but you would miss the cleanup of .docs
 159 [01:46] <jawnsy> quite conveniently, dpkg provides a "--contents" command
 160 [01:46] <gwolf> dam: Of course. But for local builds, that's often enough :)
 161 [01:46] <jawnsy> so that we can basically list the files inside our binary, and more importantly see *where* files will be installed
 162 [01:47] <gwolf> (you'd even miss the warm feeling from having a human look at it)
 163 [01:47] <jawnsy> ok, so now, please find the .deb file which was built
 164 [01:47] <jawnsy> and execute the command: dpkg --contents *deb
 165 [01:47] <jawnsy> you should get a listing of files which looks a lot like some 'ls' output
 166 [01:48] <jawnsy> this way, you can see exactly where things will install, without actually installing the binary. note that this works with any Debian (or Ubuntu) package :-)
 167 [01:48] <gwolf> ...or, conveniently, just debc *deb
 168 [01:48] <jawnsy> the Debian Perl philosophy is very much like the Perl philosophy -- there's more than one way to do it!
 169 [01:49] <jawnsy> So at this time we'd like to ask the floor for any questions they might have
 170 [01:49] <jawnsy> by now you've packaged your first Perl module, and I bet you're thinking, wow, that was easy :-)
 171 [01:49] <gwolf> agree :)
 172 [01:50] <jawnsy> in the Debian pkg-perl team we have more scripts which help us with this sort of thing, but right now you've got enough know-how to build Perl modules on your local machine
 173 [01:50] <gwolf> ..and soon afterwards, we will stat talking about how to do this in a better organized way, how to contribute to your distribution, and how to be rich and famous! (or at least, how to get to travel quite a bit)
 174 [01:50] <nhandler> How would I go about getting this module uploaded to Debian and maintained by pkg-perl ?
 175 [01:50] <nhandler> ;)
 176 [01:50] <jawnsy> That, is quite a good question.
 177 [01:51] <jawnsy> OKay let me dispel some myths first of all
 178 [01:51] <jawnsy> Many people seem to think that in order to contribute to Debian or Ubuntu, you've got to be a Debian Developer, or a Ubuntu Developer
 179 [01:51] <jawnsy> the pkg-perl team is a great way to get involved in both Debian and Ubuntu
 180 [01:51] <gwolf> good!
 181 [01:52] <jawnsy> and best of all, you don't need to be a full-fleged Developer
 182 [01:52] <gwolf> In fact, it is a great way to check how your packaging is, and have other people do the heavy lifting
 183 [01:52] <gwolf> ...Our team, as I said earlier, has around 15 active members (from a much larger list).
 184 [01:52] <jawnsy> Basically, under the pkg-perl team, we maintain all our packages as a group
 185 [01:53] <jawnsy> It's nice because if you ever get stuck, you have a great support network of people to ask for help
 186 [01:53] <gwolf> Of course, to be able to become active in it, you should at least know the basics of Perl and Debian packaging, and be interested and willing to work on any potential bugs
 187 [01:53] <jawnsy> And, more importantly, it's never hard finding a sponsor -- many Ubuntu and Debian developers are part of the group :-)
 188 [01:53] <gwolf> (hold a second for me, we are being moved to a different laboratory)
 189 [01:53] <jawnsy> maddeth: how did your build go?
 190 [01:53] <maddeth> flawlessly thankyou
 191 [01:54] <maddeth> just running a few things through my head :)
 192 [01:54] <jawnsy> Please folks, if you have questions, feel free to ask. And if you're packaging something for your own use later and you come across something else, feel free to join our group channel and ask us
 193 [01:54] <jawnsy> the pkg-perl group sits on IRC at irc.debian.org (OFTC) #debian-perl
 194 [01:55] <gwolf> ready
 195 [01:55] <jawnsy> If you're looking for a way to really make your mark on Debian or Ubuntu, please join the group. You don't need any prior experience to join the group
 196 [01:55] <gwolf> ok... So in order to work on this group, there is a couple of things to add to your skillset
 197 [01:55] <gwolf> First of all, as I said, we collectively maintain ~1300 packages
 198 [01:56] <gwolf> The best (only?) way we can do this is to coordinate it via a version control system
 199 [01:56] <gwolf> that ensures we all share each other's package modifications - And we use SVN (Subversion) for it
 200 [01:57] <gwolf> Before anybody asks - there has been talk regarding how to move our workflow towards the much more agile and flexible Git
 201 [01:57] <gwolf> ..We will get to it, sooner or later.
 202 [01:57] <jawnsy> having things in subversion or any VCS really.. is indispensible
 203 [01:57] <jawnsy> for example I could ask gwolf to take a look at something for me and help me figure out why it's not building
 204 [01:57] <gwolf> And taht also allows us to do transversal, large-scale updates
 205 [01:58] <gwolf> i.e. if we were to decide to, say, change some part of our build process, we could run it from one single place, for all of our packages
 206 [01:59] <gwolf> of course, we would have to do each change separately (i.e. via a script), but it would be grouped into one logical entity - into a single commit.
 207 [01:59] <gwolf> Anyway - where to look for this information?
 208 [01:59] <jawnsy> Seriously, if you've got some free time and a passion to learn something new, I would really recommend joining the group
 209 [02:00] <gwolf> Well, the main pointer holder is at the group's (very basic!) page: http://pkg-perl.alioth.debian.org/
 210 [02:00] <gwolf> This page just holds pointers to our inside bits
 211 [02:00] <gwolf> I would read them (and refer to them) in +- the following order:
 212 [02:00] <jawnsy> The great thing about Open Source is that if you want something, you can just dive in and do it yourself
 213 [02:01] <jawnsy> I myself have 91 packages in Debian's main repository, and like I said, I'm not a Debian Developer :-)
 214 [02:01] <gwolf> - How does this group use Subversion? http://pkg-perl.alioth.debian.org/subversion.html   (includes a short guide on a great tool we have, which is indispensable for our work: svn-buildpackage)


CategoryPackaging