Ubuntu Wiki

Dev Week -- Packaging Perl Modules -- jawnsy and Debian pkg-perl team members -- Mon Aug 31st, 2009

UTC

(03:04:24 PM) jawnsy: I'll introduce myself and a few of the people that are joining me today for this session
(03:04:47 PM) jawnsy: my name is Jonathan Yu, I'm a new(ish) member joining the pkg-perl team, and I'm not a Debian Developer
(03:05:19 PM) jawnsy: I have with me today some other members of the team: gregoa, jackyf, mogaal, Ryan52
(03:06:22 PM) jawnsy: gregoa is a Debian Developer and is up near the top rankings in terms of number of packages maintained :-) If you've ever used a Perl package on Debian, it's likely gregoa's had a hand in it at some point or another
(03:06:50 PM) jawnsy: mogaal and jackyf are both newer members to the team, so they will be giving some more insight into what it's like to begin packaging Perl modules
(03:07:20 PM) jawnsy: Ryan52 is almost a Debian Developer himself, and also contributes significantly to the group, and has also worked on the pkg-ruby-extras team to package Ruby libraries, so he can speak to the difference between the two worlds
(03:07:49 PM) jawnsy: in order to follow along, I'll ask that everyone who wishes to participate (follow along with a simple module we'll build) to install the following packages: devscripts lintian dh-make-perl
(03:08:37 PM) jawnsy: as usual, please ask questions in #ubuntu-classroom-chat, please hilight jawnsy-home and/or prefix it with QUESTION so I can see it easily
(03:09:29 PM) jawnsy: to begin with, we'll look at a CPAN module called Locale::Msgfmt, because it's simple and relatively easy to package
(03:10:04 PM) jawnsy: to begin every package, we use a tool called dh-make-perl
(03:10:17 PM) jawnsy: its purpose is to download a module and set up the skeleton framework for getting it built
(03:10:55 PM) jawnsy: Perl/CPAN developers have agreed upon a standardized toolchain which makes building, testing and installing packages a more-or-less consistent affair -- most packages, and all of the popular packages, build in the same manner
(03:11:08 PM) jawnsy: this makes it easy for us to build these modules in Debian
(03:11:25 PM) jawnsy: so to begin, type a line like this in a shell, after having installed the aforementioned prerequisite modules (that is, devscripts, lintian and dh-make-perl)
(03:11:55 PM) jawnsy: you might want to do this in a temporary folder, as the build process installs stuff in your current working directory
(03:12:02 PM) jawnsy: so I've done:
(03:12:03 PM) jawnsy: mkdir tmp
(03:12:04 PM) jawnsy: cd tmp
(03:12:14 PM) jawnsy: dh-make-perl --cpan Locale::Msgfmt
(03:12:37 PM) jawnsy: now, what that does is simply retrieve the Locale::Msgfmt package from CPAN and set up the main framework for it
(03:13:14 PM) jawnsy: so after seeing lots of text scroll by on your screen, you should end up with a .tar.gz and a directory named Locale-Msgfmt-0.14
(03:13:43 PM) jawnsy: the tarball is the upstream source, and the directory contains the upstream source plus some debhelper and other Debian-related metadata files, which are used during the build process
(03:14:11 PM) jawnsy: let's look at the anatomy of this package. taking a file listing in the Locale-Msgfmt-0.14 directory, we get this output:
(03:14:17 PM) jawnsy: bin  Build.PL  Changes  debian  dev  lib  Makefile.PL  MANIFEST  META.yml  README  t
(03:14:40 PM) jawnsy: all of those files come from the upstream source, with the exception of the debian/ subdirectory, which, as mentioned, contains all of the files that do the magic of building the module
(03:15:46 PM) jawnsy: now... I'll digress a bit from the main topic to mention that you can build these CPAN modules and install them via dpkg by using some command line parameters to dh-make-perl; namely: dh-make-perl --install --cpan Locale::Msgfmt
(03:16:22 PM) jawnsy: this is great if you are just doing some small packages which you'd otherwise have installed via CPAN anyway, but when making them for Debian, there are a few more things we need to look through to ensure good Quality Assurance
(03:17:12 PM) jawnsy: take note that while this installs the package, it won't be able to update it, so you'll be stuck at that version indefinitely. thus, for packages you're likely to use a lot, a better solution is to file a Request For Package bug, and have that package officially supported in Debian
(03:17:43 PM) jawnsy: Okay, so let's look at some of the files in debian/:
(03:17:52 PM) jawnsy: changelog  compat  control  copyright  liblocale-msgfmt-perl.docs  liblocale-msgfmt-perl.examples  rules  watch
(03:18:37 PM) jawnsy: these files all have a little bit of magic to them -- we'll come back to this later
(03:18:58 PM) jawnsy: let's first get our Perl module to build into a familiar-looking .deb
(03:19:12 PM) jawnsy: so, change back to the Locale-Msgfmt-0.14 root directory
(03:19:35 PM) jawnsy: the packages we've installed so far should include the "debuild" program, so from the main tree, simply type that on your command line:
(03:19:38 PM) jawnsy: $ debuild
(03:20:05 PM) jawnsy: you'll notice it outputs a lot of stuff, including the familiar (if you've ever installed a package via CPAN) test output
(03:20:41 PM) jawnsy: if we look a directory above us (this is why I said it's useful to create a temporary directory first).. we see a bunch of files now
(03:20:46 PM) jawnsy: so let's look at what each of these files are
(03:21:00 PM) jawnsy: liblocale-msgfmt-perl_0.14-1_all.deb  liblocale-msgfmt-perl_0.14-1.dsc           liblocale-msgfmt-perl_0.14.orig.tar.gz
(03:21:03 PM) jawnsy: liblocale-msgfmt-perl_0.14-1.diff.gz  liblocale-msgfmt-perl_0.14-1_i386.changes  Locale-Msgfmt-0.14
(03:21:19 PM) jawnsy: as Daniel Holbach (dholbach) mentioned earlier... the .deb is the binary that you actually install
(03:21:34 PM) jawnsy: so if you would like to install the package now, you can simply type: sudo dpkg --install *.deb
(03:22:12 PM) jawnsy: then there is the upload description (dsc) file, and also the original tarball, and the gzipped diff which indicates what changes have been made for Debian (notably, the inclusion of all the debian/ files)
(03:22:25 PM) jawnsy: okay, so now let's look into what the .deb would actually install in our system
(03:22:49 PM) jawnsy: thankfully, dpkg has a flag that lets us do this easily. simply:
(03:22:51 PM) jawnsy: $ dpkg --contents *deb
(03:23:08 PM) jawnsy: please, do let me know if I'm going too quickly or if there is anything unclear. do so in #ubuntu-classroom-chat.
(03:23:29 PM) jawnsy: so now we see a file listing, which includes exactly where everything will be installed on your system (if you do dpkg --install) on that package
(03:24:20 PM) jawnsy: now, in Debian we don't want to waste users' space
(03:24:34 PM) jawnsy: so we look at things like the examples and the documentation to make sure they are really useful, before installing them in /usr/share/doc
(03:25:05 PM) jawnsy: so for example in this case, the README file doesn't tell us anything useful
(03:25:24 PM) jawnsy: this is a good time to bring up what some of those magic files are
(03:25:51 PM) jawnsy: the README file is installed because it is listed in this file: liblocale-msgfmt-perl.docs
(03:26:03 PM) jawnsy: similarly all the examples listed in liblocale-msgfmt-perl.examples are installed
(03:26:12 PM) jawnsy: so if there is ever documentation or examples you don't think should be installed, you can remove them here
(03:26:22 PM) jawnsy: so let's look inside the *.docs file
(03:26:34 PM) jawnsy: we find a single line, README, which is the name of the file that is installed, that we don't want
(03:26:48 PM) jawnsy: if we simply remove that line, then the debhelper installation mechanism will not install the file
(03:27:07 PM) jawnsy: now, since an empty file is useless, we can just remove the file altogether
(03:27:36 PM) jawnsy: changing back to the main Locale-Msgfmt-0.14 root directory, let's try rebuilding this package
(03:27:46 PM) jawnsy: by running `debuild' again
(03:28:14 PM) jawnsy: so again, the same hubbub of text scrolling by
(03:28:30 PM) jawnsy: and we can examine the contents once again (dpkg --contents *deb), noticing this time that the README is no longer installed
(03:29:03 PM) jawnsy: so we'll look briefly at the function of the other files
(03:29:13 PM) jawnsy: and then I'll open the floor for some questions and answers
(03:29:38 PM) jawnsy: Perl packaging really isn't difficult, and being a Perl user/developer myself, I got into it initially to get some packages I needed into Debian
(03:29:59 PM) jawnsy: I should mention a great thing about the pkg-perl group is that we have 10+ members, including gregoa, who are prolific Debian Developers
(03:30:25 PM) jawnsy: and it's often a matter of days or weeks to get a package uploaded, compared to months as you'd have with the normal mentors process
(03:30:50 PM) jawnsy: in the context of Ubuntu, we also handle our modules which have been sync'd to Ubuntu, and we have two important liasions to the Ubuntu community (many more are welcome!)
(03:31:07 PM) jawnsy: nhandler aka Nathan Handler, and Iulian Udrea are both members of the team
(03:31:23 PM) jawnsy: everything that benefits Debian or Ubuntu, beenefits both -- the changes flow both ways, and that is really the magic of open source
(03:31:34 PM) jawnsy: anyway, back to the meaning of the rest of the debian/ files
(03:31:47 PM) jawnsy: after building I've now got something like this in debian/:
(03:31:49 PM) jawnsy: changelog  control    files                  liblocale-msgfmt-perl.debhelper.log  liblocale-msgfmt-perl.substvars  watch
(03:31:52 PM) jawnsy: compat     copyright  liblocale-msgfmt-perl  liblocale-msgfmt-perl.examples       rules
(03:32:13 PM) jawnsy: the .log file, liblocale-msgfmt-perl and the .substvars are just leftover files from the build, and aren't necessary
(03:32:48 PM) jawnsy: if they bother you, you can have the package build cleaned up by changing to: Locale-Msgfmt-0.14 again, and issuing:
(03:32:52 PM) jawnsy: $ fakeroot debian/rules clean
(03:32:57 PM) jawnsy: (though you'll need to install fakeroot for that)
(03:33:15 PM) jawnsy: (that command might work as debian/rules clean, without fakeroot, I'm not sure)
(03:33:40 PM) jawnsy: oh, heh, I'm learning new things every day. Ryan52 mentions to me out of band that "debclean" also accomplishes this task
(03:34:19 PM) jawnsy: okay, so the remaining files--
(03:34:49 PM) jawnsy: changelog is the Debian changelog file, which lists things that have been done in the Debian package only (upstream packages generally, but not always, also include their own changelog for the purpose of CPAN users)
(03:35:11 PM) jawnsy: from the dpkg contents listing we had before, recall that we saw these files:
(03:35:14 PM) jawnsy: -rw-r--r-- root/root      1063 2009-07-09 05:16 ./usr/share/doc/liblocale-msgfmt-perl/changelog.gz
(03:35:18 PM) jawnsy: -rw-r--r-- root/root       163 2009-08-31 11:09 ./usr/share/doc/liblocale-msgfmt-perl/changelog.Debian.gz
(03:35:24 PM) jawnsy: the changelog.gz is the upstream package changelog
(03:35:33 PM) jawnsy: the .Debian file is the one we know as debian/changelog
(03:36:12 PM) jawnsy: the changelog is also the source of our version number tracking for packages
(03:36:26 PM) jawnsy: you'll notice in the changelog two lines; the first and the last:
(03:36:26 PM) jawnsy: liblocale-msgfmt-perl (0.14-1) unstable; urgency=low
(03:36:32 PM) jawnsy:  -- Jonathan Yu <jawnsy@cpan.org>  Mon, 31 Aug 2009 11:09:19 -0400
(03:36:34 PM) jawnsy: which are notable
(03:36:56 PM) jawnsy: now, the first one is the package name + version number; unstable is the release name in Debian, where all new packages go
(03:37:11 PM) jawnsy: the trailer has, importantly, my name and e-mail address
(03:37:37 PM) jawnsy: this is what gets recorded as the "uploader" of a given version of a package, even though you won't be uploading packages directly -- a Sponsor does that on your behalf
(03:37:52 PM) jawnsy: though if you're a Ubuntu Developer then you'd be the uploader and wouldn't need a sponsor (but you already know that)
(03:37:58 PM) mode (+o dam ) by jawnsy
(03:38:26 PM) jawnsy: the next file, control, is where most of the magic happens
(03:38:49 PM) jawnsy: its purpose is to specify things like what the package is (ie a description), and other things like the dependencies the package needs
(03:39:16 PM) jawnsy: importantly, Build-* things [note: the first paragraph relates to the Source package, from which binaries are built]
(03:39:32 PM) jawnsy: the Build-* relationships tells us what we need to install in order to build something, though that is separate from what we need to use the module
(03:39:51 PM) jawnsy: so, for example, while I might need Test::Exception to run tests during building, I don't need that in the binary package
(03:40:05 PM) jawnsy: this explains why some packages will show: libtest-exception-perl in Build-Depends, but not in Depends
(03:40:30 PM) jawnsy: I'll leave the exact meaning of the rest of the fields as an exercise for you, but the Debian Policy Manual describes them all at length
(03:40:58 PM) jawnsy: the copyright file contains information relating to the copyright of all our packages, which is important in Debian and Ubuntu because that is how we protect free software
(03:41:30 PM) jawnsy: the Debian Free Software Guidelines are a central part of the Debian Social Contract, and I imagine that to a great extent the Ubuntu community agrees
(03:41:59 PM) jawnsy: so moving along, the *.examples file is just like *.docs (which we removed).. and it contains a list of places to find examples
(03:42:09 PM) jawnsy: in this one, we find one line: t/samples/*
(03:42:36 PM) jawnsy: which does what you'd expect with a shell glob, it just gets all the files in that directory and installs them (explaining much of what you saw in the dpkg contents listing)
(03:42:55 PM) jawnsy: now there are two more files left to explain
(03:42:59 PM) Ryan52 is now known as |Ryan52_
(03:43:04 PM) jawnsy: the watch file is autogenerated and most often doesn't need to be modified
(03:43:17 PM) |Ryan52_ is now known as |Ryan52
(03:43:26 PM) jawnsy: the purpose of this file is to scan the upstream for new releases, and given that most Perl-related things are released via CPAN, you probably won't need to touch this file
(03:43:41 PM) jawnsy: the rules file is an important part of the build process
(03:44:02 PM) jawnsy: currently the format is just a simple makefile, but it calls the debhelper build system to get everything done
(03:44:36 PM) jawnsy: Locale-Msgfmt happens to be a simple module as I mentioned before
(03:44:39 PM) jawnsy: so the rules file just contains:
(03:44:40 PM) jawnsy: %:
(03:44:43 PM) jawnsy: dh $@
(03:44:50 PM) jawnsy: and debhelper does the other magic :-)
(03:45:07 PM) jawnsy: there are some other features of debhelper which we use from time to time, but this gives you a basic idea of how to build Perl modules, and what each file is for
(03:46:05 PM) jawnsy: before I open the floor up to some questions, do you guys from the pkg-perl team have any other comments to make?
(03:46:49 PM) jawnsy: ...? :-)
(03:47:00 PM) jawnsy: I guess they're happy with my treatment of this :P
(03:47:24 PM) jawnsy: I could go into a more complicated example, or rant about why the pkg-perl team is cool, or answer other questions
(03:48:23 PM) jawnsy: I hope that this session (finished mostly in under 40 minutes) shows you how easy it is to participate in the team
(03:48:37 PM) jawnsy: as I mentioned, I'm not a Debian Developer nor Ubuntu Developer
(03:48:49 PM) jawnsy: yet I can contribute to both projects through the group and with the help of the many DDs that sponsor my uploads
(03:49:09 PM) jawnsy: dinxter asks:
(03:49:09 PM) jawnsy: QUESTION: Is there somewhere i can look for a reference for what all those magic debelper files like .example are for and how they work, .install, etc
(03:49:44 PM) |Ryan52: the debhelper manpages are good. "man dh", "man dh_installexamples", etc.
(03:50:09 PM) jawnsy: you'll notice to when you run debuild, you'll get a list of a bunch of debhelper commands that are being executed
(03:50:11 PM) gregoa: and for the big picture: man debhelper
(03:50:25 PM) jawnsy: the scripts are usually named according to what they do, so it's not too difficult
(03:50:35 PM) jawnsy:    dh_installman
(03:50:35 PM) jawnsy:    dh_installcatalogs
(03:50:35 PM) jawnsy:    dh_installcron
(03:50:38 PM) jawnsy: ^ some example output
(03:50:53 PM) jawnsy: so if one is curious about what those do, you can take a look at their manpages
(03:51:18 PM) jawnsy: there aren't too many, and if you join the group (as we hope you will), it's easy to ask for help
(03:52:35 PM) jawnsy: I myself have only really packaged Perl modules, but many of the concepts are the same -- what the meta-files do, getting familiarzed with Debian and Ubuntu's social policies, etc
(03:52:58 PM) jawnsy: so that means it is a great way to begin contributing to Debian or Ubuntu, prior to even becoming a * Developer :-)
(03:53:26 PM) jawnsy: alexm mentions that even though I've gotten everyone to install lintian, I haven't explained what it does, or what the warnings mean
(03:53:43 PM) |Ryan52:  
(03:53:56 PM) jawnsy: lintian is Debian's package checking system. it's a Perl program with many plugins and checks that looks at your code to figure out possible places you might have done something incorrectly
(03:54:10 PM) jawnsy: this is another tool which ensures Debian and Ubuntu's quality assurance
(03:54:39 PM) jawnsy: we can run this manually from the main directory (where our .deb file is)
(03:55:11 PM) jawnsy: so I get this output:
(03:55:13 PM) jawnsy: aven'jon(~/tmp)> lintian *changes
(03:55:13 PM) jawnsy: W: liblocale-msgfmt-perl: script-with-language-extension usr/bin/msgfmt.pl
(03:55:16 PM) jawnsy: E: liblocale-msgfmt-perl: description-synopsis-is-duplicated
(03:55:19 PM) jawnsy: W: liblocale-msgfmt-perl: description-contains-dh-make-perl-template
(03:55:37 PM) jawnsy: there are flags you can use to get more descriptive output, which also includes what the issue is and a proposed way of fixing it
(03:55:58 PM) jawnsy: lintian is usually very good at doing this, though it is no substitute for experience (and this is where mentors and the sponsors come in)
(03:56:23 PM) jawnsy: first off is the script being installed with a language extension. Debian policy does not like .pl files being installed in /usr/bin
(03:56:41 PM) jawnsy: which is where that warning comes from. we can fix this by providing an override during the install process
(03:56:47 PM) jawnsy: let me mention a bit about how that works
(03:56:54 PM) gregoa: I always run lintian as "lintian -iI --pedantic --color=auto <pkg>.changes" get also get the informational and pedantic messages and the nice informations jawnsy mentioned
(03:57:06 PM) jawnsy: when you're building the package, by default it puts it somewhere like your home directory or in debian/
(03:57:28 PM) jawnsy: in our case, that's what produced the debian/liblocale-msgfmt-perl folder
(03:57:31 PM) jawnsy: prior to us cleaning it
(03:57:41 PM) jawnsy: that is the "staging area" where things are installed, before being rolled into the debian binary
(03:58:01 PM) jawnsy: so, in an override we'd be able to rename the script in this staging area, which thus changes the name that it's installed as, so as to comply with policy
(03:58:14 PM) jawnsy: overrides are just a way for us to change the default behaviour of debhelper, and it's part of what makes it so flexible and great :-)
(03:58:45 PM) jawnsy: this is all probably a bit complicated for the beginner, but that's how that issue is tackled
(03:58:53 PM) jawnsy: we only have a minute or two left so I'll mention the other two warnings
(03:59:04 PM) jawnsy: one of them comes from what dh-make-perl inserts in your files, to make sure you actually edit them
(03:59:19 PM) jawnsy: you'll see the boilerplate text when looking at the files and remove them, so you won't get that warning
(03:59:31 PM) jawnsy: the synopsis being duplicated has to do with a bad description in the description field
(03:59:45 PM) jawnsy: notably, the first line of the Description field (which is our synopsis or short description of the module) is the same as our long description
(04:00:15 PM) jawnsy: so usually this is where a packager would need to describe what the package does in a few lines (a paragraph or two at minimum) so that users know what it is :-)
(04:00:27 PM) jawnsy: so this about concludes our talk about Debian/Ubuntu Perl Packaging
(04:00:55 PM) jawnsy: I do hope you learned something from this, and that you consider joining the group, or even just visiting to see if it's something you might be interested in
(04:01:03 PM) jawnsy: we are on irc.debian.org (OFTC) in #debian-perl
(04:01:04 PM) jawnsy: aven'jon(~/tmp)> lintian *changes
(04:01:04 PM) jawnsy: W: liblocale-msgfmt-perl: script-with-language-extension usr/bin/msgfmt.pl
(04:01:07 PM) jawnsy: E: liblocale-msgfmt-perl: description-synopsis-is-duplicated
(04:01:08 PM) jawnsy: oops wrong paste
(04:01:10 PM) jawnsy: err
(04:01:24 PM) jawnsy: this page welcomes new members, and provides lots of useful information: http://wiki.debian.org/Teams/DebianPerlGroup/Welcome
(04:01:38 PM) jawnsy: thank you all for your
(04:01:39 PM) jawnsy: time
(04:01:42 PM) jackyf: as a sort of newscomer, I can add that the atmosphere of pkg-perl IRC discussion channel, where is significant part of collaboration is done, is very warm and, so, easy to join :)
(04:01:53 PM) Riddell: thanks jawnsy.  In a couple of minutes me and agateau will be doing an introduction to Plasmoid with Python
(04:01:55 PM) jawnsy: now I shall surrender the floor to agateau and Riddel for "Fun with Python Plasmoids" :-)