2005-12-10

Revision 37 as of 2008-08-06 16:24:21

Clear message

This page is kept for historic reasons and as a log file of the MOTU/School session, for more general Packaging information, please check out PackagingGuide/Basic.

Note: The transcribbler of this page doesn't know squat about packaging. So this should be reviewed by someone who actually knows what it's talking about. ;-)
Editors of this page would do well to refer to http://netz.smurf.noris.de/logs/freenode/2005/12/10/%23ubuntu-motu-school.log for the source text.

Suggested prerequisite reading: The Ubuntu Packaging Guide

The Debian System includes a whole chapter on packaging, which starts out doing it manually (without cdbs/debhelper/...). Maybe you consider adding this info below, just in case someone is interested. --martin

Packaging without debhelper and/or CDBS

Presented by Andrew Mitchell (ajmitch), December 10, 2005 17:00-19:00 UTC, IRC channel #ubuntu-motu-school

About Andrew Mitchell:

Andrew is Debian Developer and MOTU since January 2005. Actually to become a Debian Developer (abbrev.: DD) you will start as “Padawan”, so you learn packaging from the ground. To speak of a “Master of the Universe” (abbrev.: MOTU) he is one of our heros in several transitions and merge runs. He knows what he is doing and is always very sensible to do no mistakes in his packages.

His opinions and views are appreciated all over the MOTUs.

\sh

HELLO

The Channel is now switched to "Moderated" Status...so nobody can write something, only the moderators and operators. I like to welcome all of you to our first lecture about Debian Packaging. Today, Mr. Andrew Mitchell will show you, how to create debian packages without the help of debhelper and cdbs. If you have any questions about it, please query siretart or \sh, we will collect the questions and will raise them towards Mr. Andrew Mitchell. OK..I'll give now the voice to Mr. Andrew Mitchell aka ajmitch.


ajmitch:

Thanks, \sh

Welcome, everyone, to our newly created MOTU school.

What I'll be giving today, is not a packaging howto as such but a walk through creating a debian package without the use of dh_make, debhelper, or cdbs. I have done this example package in those 3 ways to quickly demonstrate the differences, also. We might as well start with looking at an example package which I very quickly prepared yesterday.

On http://ajmitch.dyndns.win.co.nz/debuild/ubuntu/school/scratch/ there are 2 files to be fetched if you want to follow along on your own box. Otherwise, I have the core part on pastebin Smile :) Please fetch the upstream tarball from http://www.southern-storm.com.au/download/treecc-0.3.6.tar.gz rather than from my box Smile :) (Any questions, please ask via \sh or siretart.)

The 2 files to fetch from my system are treecc_0.3.6-0ubuntu1.diff.gz and treecc_0.3.6-0ubuntu1.dsc.

siretart copied the source package to http://tiber.tauware.de/~siretart/treecc/ - might be faster for European and American users.]
thanks siretart

If you're wanting to unpack this source package, please install dpkg-dev if you haven't done so already.

If haven't fetched the files yet:

wget http://www.southern-storm.com.au/download/treecc-0.3.6.tar.gz \
http://ajmitch.dyndns.win.co.nz/debuild/ubuntu/school/scratch/treecc_0.3.6-0ubuntu1.d{sc,iff.gz}

Ok, if you've got those files in a directory now..

dpkg-source -x treecc_0.3.6-0ubuntu1.dsc

In order to unpack them. I'll quickly explain about this after I remember to tell you to rename a file Smile :)

mv treecc-0.3.6.tar.gz treecc_0.3.6.orig.tar.gz

before you try the dpkg-source step Smile :)

A debian (or ubuntu) source package is usually made up of upstream source and changes that we have made. Those changes include packaging metadata which say how the package should be built, scripts to run on installation, etc. We keep the upstream tarball as-is, and the debian changes are kept as a single compressed patch against the upstream source. (Please feel free to send myself or \sh, siretart a msg if you want to interrupt at some point.) The treecc_0.3.6-0ubuntu1.dsc file that was downloaded has the tarball and the diff.gz listed in it, along with the md5sums of them. dpkg-source uses this to apply the patch.

Now if we go into the treecc-0.3.6/ directory that was created, we see a fairly normal unpacked tarball, except that we now have a debian/ directory which contains the relevant packaging info. In the debian/ dir there are 4 main files I want to talk about now.

The changelog file has a list of the changes made, both fixes we've made to upstream source and packaging changes. As I have just done a single revision, I have 1 changelog entry which is not very informative Smile :) The line:

treecc (0.3.6-0ubuntu1) dapper; urgency=low

has some important info. It has the version number, as upstream-debianrevision. It has the distribution (dapper), and the urgency field, which is not used in ubuntu, as far as I'm aware. For ubuntu, we number new packages, or new upstream versions with a debian revision of -0ubuntuX, to avoid clashes with debian in the future. A fix to a package 1.2.3-4 would be 1.2.3-4ubuntu1, BTW. After this line, changes are listed, in the format shown and the changelog entry is closed off by the name, email and date.

Now, onto debian/control. Quoting from the debian policy manual..

  • The debian/control file contains the most vital (and version-independent) information about the source package and about the binary packages it creates.

    The first paragraph of the control file contains information about the source package in general. The subsequent sets each describe a binary package that the source tree builds.

In this case, we build 1 binary package, so there's only 2 paragraphs of info (source+binary). dh_make would kindly fill this out for you from a template. (BTW we accept questions at any time Wink ;) )

Debian policy is a very good reference for what we need, but it is not intended as a HOWTO.

\sh

and we are answering them directly...

http://www.us.debian.org/doc/debian-policy/ch-controlfields.html#s-sourcecontrolfiles has the relevant info about what fields are required.

A question about finding what to put in the Build-Depends field - much of that is done by reading what the package needs in its README, configure output, etc.

SWAT

what does the ${shlibs:Depends} stand for?

ajmitch will work through the fields & explain them :)

I'll go through the source package fields first.. & I'll do a bit of quoting from the policy document to explain. Smile :)

  • Source: This field identifies the source package name. Probably one of the easiest for us to fill in.

  • Section: This is the debian section that the package comes under. There's a list in debian policy, or we'd look at a similar package to see what section it is in.

  • Priority: is used to sort the archive into varying priorities, such as required, standard, optional. It's mainly used in debian & in main, as most of universe would be optional or extra.

  • Maintainer: This is the name & email address of the person who created or looks after the package. We generally don't change this in ubuntu if we're making fixes to a debian package but a new package, we set our name & email here.

  • Build-Depends: came up earlier - it's a list of packages that are required to build this from source. the exception is build-essential, which is assumed to be installed when building any package. For this package, I read the README file & checked what configure needed in order to build properly.

  • Standards-Version: This is the version of debian policy that the package conforms to - there's a list of changes, but it's expected that we follow the latest. the latest is 3.6.2.1, but the .1 is for editorial changes, so it can be left off in this file.

Now... onto the binary sections Smile :)

ajmitch will have to try & speed this along

Feedback is welcome to tell us if this is going too slow, too fast, etc.

retrix

how do you find out what the latest standards version is?

aha

dpkg -l debian-policy

gives you the version of the installed policy package debian-policy is also available online, from http://www.debian.org/doc/debian-policy/

ok, onwards..

  • Package: this gives the name of the *binary* package that gets created.

  • The Architecture: field indicates what arches the packages should build on. It's common to specify all, for a package that is architecture-independent, eg python code or any, for a package that compiles on any architecture.

  • The Depends: field gives info about what binary packages this one will depend on when installing - what it needs to run, rather than what it needs to compile.

    ${shlibs:Depends} gets replaced when the package is built to give info about what libraries it needs.

  • The Description: field is 2 parts - a short, 1 line description which doesn't need to be a full sentence and a paragraph or so of text about what it is.

A question about "architecture: any" vs. "architecture: all"

Packages that are "arch: all" get built once, and work on all architectures - they are usually data files, python code, shell scripts, etc. "arch: any" gets compiled for any arch - you could list i386 amd64 powerpc instead to limit it to these archs.

ajmitch would like to move onto the big debian/rules if there are no objections, since this is taking a little longer than I expected :)

siretart hasn't received any question yet, feel free to ask anything which you think that might be noteworthy

debian/rules is the main file that controls how everything gets built - it is a makefile, with specific required make targets. As an example, I have the same package done 3 ways (debhelper, cdbs, without helper programs) to compare. Smile :)

170 lines for this debian/rules
61 lines with debhelper
and 12 with cdbs

You can probably tell that this way is more verbose & more complex Smile :)

Again, from debian policy..

  • This file must be an executable makefile, and contains the package-specific recipes for compiling the package and building binary package(s) from the source.

    It must start with the line "#!/usr/bin/make -f", so that it can be invoked by saying its name rather than invoking make explicitly.

    Since an interactive debian/rules script makes it impossible to auto-compile that package and also makes it hard for other people to reproduce the same binary package, all required targets MUST be non-interactive. At a minimum, required targets are the ones called by dpkg-buildpackage, namely, clean, binary, binary-arch, binary-indep, and build. It also follows that any target that these targets depend on must also be non-interactive.

Basically, it says that we need these few makefile targets, and they can't ask the builder anything since everything uploaded in ubuntu gets built on a build daemon, unattended.

We might as well look at the debian/rules now.

Firstly, I started off by setting up some commonly used variables. The DEB_BUILD_OPTIONS environment variable can control some aspects of building, often for things like not stripping a binary. Some small macros are defined to check that we are in the right directory & have uid 0. fakeroot is often used to fake the existence of uid 0 when building, since packages can & should be built as a user. DESTDIR is important, as it's an option passed to make to tell it where to relocate the installed files to when running make install. So it's used throughout, in the other directory variables.

We have listed 2 stamp rules, which are used to avoid repetition. make was originally meant for building plain source, so it needs some files to be created to avoid running targets again.

  • The configure-stamp: target is what contains the call to configure. In this case, we're not passing many configure options, but some packages do take quite a few, which can be put in here.

  • The build-stamp: target has configure-stamp after it, to indicate that it needs configure-stamp to run first, before building. It first checks that we're in the right directory, and runs make. The debian rules section is what is required by debian policy.

  • build: just depends on build-stamp, and does nothing more - if build-stamp was run it won't repeat it.

  • The clean: target gets rid of everything, and restores things to a (relatively) pristine state. if patches had been applied, they would be unapplied here. stamp files are removed, and make distclean is run to get rid of the build output.

  • binary-indep: is another fairly empty rule, which just depends on build. If this package was Architecture: all, we'd be doing a lot more in here.

ajmitch now has a few lines to give options to install, to avoid typing them over & over later on

  • The binary-arch: target is now where we do most of the actual packaging of the files which is why I've commented this one the most. Smile :)

If there are any questions up till now, please ask

We have a question: "How if the distclean rule from the source Makefile doesn't clean properly the source? Is there a way to restore it in the clean: rule?"

If distclean doesn't properly clean things, then often we resort to just using rm in the clean: rule. Upstream authors often don't have perfect makefiles that clean things. Smile :) If distclean fails (eg the makefile doesn't exist yet), then it will still continue.

LaserJock

do you have to have the stamps?

The stamp files aren't needed, but they're useful to avoid running things again, especially if it was a long compile & you just want to test out packaging changes. Then you could run the build target once & the binary-arch target multiple times to get things right. It's not as useful running in pbuilder, when we're always doing a clean build. Any other questions about that?

ok, will move on down the file Smile :)

First step is installing what we've built, into the directory DESTDIR. So you end up with a directory structure like debian/treecc/usr/bin, for example.

ajmitch chose to put make check in there, to run the program test suite. No point packaging up a broken program, so we check it where we can.

Now, what we don't see with using debhelper or cdbs is the DEBIAN directory which is debian/treecc/DEBIAN.

The control, changelog, maintainer scripts go in here - debhelper hides this from us nicely.

This next section about installing docs is handled by 1 debhelper program, in this case we just copy the files into the right place.

I'll get to the debian/copyright file after this, it's also rather important.

Now, the package scripts - for this package we have a script to run after installation, eg 'Setting up treecc (0.3.6-0ubuntu1)'. The script I have is very simple, all it does is register the info documentation. You can look at it in debian/postinst if you want to - I'll try not to spend too much time on this right now. Smile :)

The prerm script is subsequently called when removing or upgrading the package. The URL in the comments has details of how these interact.

After this, we compress the documentation - I didn't compress the README or NEWS files, although I could have here. A common thing is to also have a README.Debian file, which has information specific to the package on debian/ubuntu. I was also lazy & didn't install the other documentation & example files, apart from the manual (in info format).

Questions came up earlier about ${shlibs:Depends} in debian/control.

dpkg-shlibdeps works out what libraries are needed by the binaries, via a nice system of magic. Library packages have shlibs files to indicate the binary package version required of the library when built against the library headers. These are used & put into the debian/substvars file by dpkg-shlibdeps. There can be a number of lines in substvars, but the most common one you'll ever see or use will be ${shlibs:Depends}. Another common one used with debhelper is ${misc:Depends}.

dpkg-gencontrol will take the file from debian/control, and generates a control file suitable for the binary package in debian/treecc/DEBIAN/control. It will take those substvars & do the replacements needed - more info on that is available in the dpkg-gencontrol manpage, in the VARIABLE SUBSTITUTION section.

The next part is commonly done by dh_md5sums when using debhelper - here we have to generate md5sums manually for each file that will get installed. This is mainly used by debsums (if you choose to use it) to track modified files.

Now, the final part in this target is dpkg-deb, which does the part of taking the files in DESTDIR, and generating a binary package. It takes the files in DEBIAN & puts them in the control.tar.gz, and the rest from debian/treecc into data.tar.gz, and packs it into the ar archive. So, by the time we get here, we have a binary package built. Smile :)

any questions? Might as well allow some public questions for now. Smile :)

ajmitch hopes it hasn't been too confusing or too slow for people. ajmitch sets mode: -m


siretart

public question time!

ajmitch still wants to briefly talk about the copyright file after this
ajmitch will step out for a couple of minutes, ask questions while I'm afk :)

retrix

Whats the next step after building a package, i.e. how do we get it into universe?

Kyral

Submit to REVU for review

raphink

retrix: for that, have a look at the REVU website: http://revu.tauware.de/

nmsa

What I don't see are cdbs and debhelper. Will this follow?

raphink

nmsa: maybe in the next lecture ;)

\sh

nmsa: this will come later Smile :) we will have the cdbs maintainer of debian talking to us...

Kyral

nmsa: This was meant to show you what Debhelper and CDBS do for you

SWAT

aha :)

SWAT

they do quite a freaking lot for us

nmsa

Kyral: in comparison to make, right?

sladen has been keeping a rough-and-ready summary at: http://www.paul.sladen.org/ubuntu/tutorial/01/commands.txt

please note, I haven't quite finished yet Wink ;)

siretart

are there any questions left?

siretart

ajmitch wants to loose some words on debian/copyright

as well as getting the thing built Wink ;)

Kyral

If no one has an idea for the next lesson I could do one on PBuilders

\sh

Kyral: lets discuss this later...I'll have to make all the schedules :)

Kyral

I have to go up to the Linux Lab

siretart

what does the auditorium think about +m?

raphink

siretart: not sure it's necessary

Ok, onto debian/copyright then

You might note that the date is fairly old - that was when I first built this package with dh_make. The copyright statement in this package is a little messy. As it has the standard 3 paragraphs of GPL, with a reference to the full text and then it has some further explanation from upstream about the licensing of output of this program.

If the program has certain exceptions like this, or linking exceptions, then you need to put them in here. This should ideally have a copyright (c) xxxx-yyyy Southern Storm Software in it

Right, I think that's most of what we need for the debian/ files. If you want, you could build the package now with dpkg-build-package -rfakeroot you see what happens in debian/rules then all I can say is that it 'works for me'. Wink ;)

stephank

what's the difference when I use debuild?

shorter to type Wink ;) debuild runs dpkg-buildpackage, then lintian/linda. Then it will sign the .changes and/or .dsc file created. I always prefer to prepare a source package for upload with debuild -S (and other options where needed) since everything needs to be signed before upload.

siretart

stephank: debuild is basically a wrapper for dpkg-buildpackage. It is part of the package devscripts, and can be nicely configured in ~/.devscripts. I've set in ~/.devscripts that I do not want to sign my packages with debuild, but manually with debsign. (I consider signing packages as part of the upload process, not the build process).

ajmitch does it the other way, signing unless -us -uc is passed, but that's because I just have the signing habit.

stephank

I see, thanks. :)

This won't be uploaded to the archive, 0.3.6-2 is already in there. Smile :)

any other questions? Perhaps questions about the debhelper or cdbs options?

if you want to compare rules files Smile :)

Using debhelper is often simple, but it doesn't require that you know what is going on.

akurashy

What's cdbs?

cdbs is even simpler for a standard package. cdbs = common debian build system. It abstracts away nearly everything. So debhelper, etc are even hidden from the packager.

sistpoty

it's a set of make-rules that you can include in debian/rules

LaserJock

so is there some kind of "rule of thumb" for when to use the different systems?

Use what feels most appropriate Smile :) cdbs has its limitations, if you find yourself having to hack round things then you know that it might not be best

nmsa

what are *.mk files?

they're just makefiles

LaserJock

so would cdbs have a lot of overhead for a small packages?

sistpoty

LaserJock: yep

well, not a lot of overhead

siretart

LaserJock: the good thing about debhelper is, that if it breaks, you know excatly where it broke

treecc is a relatively small package, and it uses cdbs

siretart

LaserJock: with cdbs, you don't get that information easily

cdbs requires some serious make-fu if you need to move outside the box

siretart

LaserJock: so for small packages, cdbs might be okay. For big and complicated ones, debhelper might be easier for maintenance

It scares me, and I'm sure it scares its author. Smile :) But 90% of the time it's great for what I want.

LaserJock

so maybe for a package that doesn't require any funky stuff cdbs would be good but if you need to do quite a bit of tweaking then debhelper would be better?

yeah, or another build system

siretart

ajmitch: please dont recommend yada *g* haha

LaserJock

cdbs just scares my right now because it is too magical ;-)

slomo

LaserJock: but it safes you much time for standard packages ;)

certainly I managed to get a package done in around 5 minutes one time with cdbs Smile :) and uploaded about 30 minutes later after some MOTUs checked it Smile :)

slomo

ajmitch: hehe dito, but only because copyright was easy for this one ;)

ok, 2 hours, I guess that's probably about long enough unless there's other questions Smile :)

stephank

Perhaps a bit unrelated, but why is dpatch bad? (read that somewhere) Is it replaced by debhelper or cdbs?

stephank: dpatch is not bad, as such and it's a patch system that is used along with debhelper/cdbs. It's 'bad' in that we shouldn't be adding dpatch to packages that don't use it in debian since that just introduces intrusive changes

stephank

Okay. It has an odd patch format, IMO. :)

Odd? dpatch has a fairly standard patch format, just with a header.

minghua

another reason people consider dpatch "bad" is because dpatch doesn't have a standard debian/rules target to apply all needed patches or simple-patchsys with cdbs

LaserJock

are there other patch systems out there?

yes, there are several others

tseng

LaserJock: none of them are suitable to add to packages that dont use them.

LaserJock

tseng: I agree I just wonder about what to use when making new packages I tend towards dpatch for my packages.

tseng

dpatch or simple-patchsys for cdbs. both are plain patches. dpatch with a header

minghua

This is a main complain from security people because they need to look at and patch other people's packages quite often yes, it can be annoying as do MOTUs :)

siretart

excatly

tseng

the complaints are off topic, its enough to say "dont add a patch system to someone elses package"

minghua

If the maintainer uses dpatch and has some obscure debian/rules, it takes quite some time to figure out the correct way to get the source tree to patch against

sistpoty

debian source packages are a kind of patch system in itself, since the .diff.gz is basically a patch.

\sh

Ok...any other questions can be raised on ubuntu-motu mailing list


(Thanks and kudos followed...)


CategoryArchive