'''This page is kept for historic reasons and as a log file of the [[MOTU/School]] session, for more general Packaging information, please check out [[PackagingGuide/Basic]].''' ||<#ccffcc>Note: The transcribbler of this page doesn't know squat about packaging. So this should be reviewed by someone who actually knows what it's talking about. ;-)<
>Editors of this page would do well to refer to http://netz.smurf.noris.de/logs/freenode/2005/12/10/%23ubuntu-motu-school.log for the source text.<
><
>Suggested prerequisite reading: [[http://doc.ubuntu.com/ubuntu/packagingguide/C/|The Ubuntu Packaging Guide]]<
><
>[[http://debiansystem.info|The Debian System]] includes a whole chapter on packaging, which starts out doing it manually (without cdbs/debhelper/...). Maybe you consider adding this info below, just in case someone is interested. --martin|| = Packaging without debhelper and/or CDBS = Presented by '''Andrew Mitchell (ajmitch)''', December 10, 2005 17:00-19:00 UTC, IRC channel #ubuntu-motu-school ||About Andrew Mitchell:|| || Andrew is Debian Developer and MOTU since January 2005. Actually to become a Debian Developer (abbrev.: DD) you will start as “Padawan”, so you learn packaging from the ground. To speak of a “Master of the Universe” (abbrev.: MOTU) he is one of our heros in several transitions and merge runs. He knows what he is doing and is always very sensible to do no mistakes in his packages.<
> <
>His opinions and views are appreciated all over the MOTUs.|| '''__\sh__''' HELLO The Channel is now switched to "Moderated" Status...so nobody can write something, only the moderators and operators. I like to welcome all of you to our first lecture about Debian Packaging. Today, Mr. Andrew Mitchell will show you, how to create debian packages without the help of debhelper and cdbs. If you have any questions about it, please query siretart or \sh, we will collect the questions and will raise them towards Mr. Andrew Mitchell. OK..I'll give now the voice to Mr. Andrew Mitchell aka ajmitch. ---- '''__ajmitch:__''' Thanks, \sh Welcome, everyone, to our newly created MOTU school. What I'll be giving today, is not a packaging howto as such but a walk through creating a debian package without the use of dh_make, debhelper, or cdbs. I have done this example package in those 3 ways to quickly demonstrate the differences, also. We might as well start with looking at an example package which I very quickly prepared yesterday. On http://ajmitch.dyndns.win.co.nz/debuild/ubuntu/school/scratch/ there are 2 files to be fetched if you want to follow along on your own box. Otherwise, I have the core part on pastebin :) Please fetch the upstream tarball from http://www.southern-storm.com.au/download/treecc-0.3.6.tar.gz rather than from my box :) (Any questions, please ask via '''\sh''' or '''siretart'''.) The 2 files to fetch from my system are '''treecc_0.3.6-0ubuntu1.diff.gz''' and '''treecc_0.3.6-0ubuntu1.dsc'''. siretart copied the source package to http://tiber.tauware.de/~siretart/treecc/ - might be faster for European and American users.]<
> thanks siretart If you're wanting to unpack this source package, please install dpkg-dev if you haven't done so already. If haven't fetched the files yet:<
> {{{ wget http://www.southern-storm.com.au/download/treecc-0.3.6.tar.gz \ http://ajmitch.dyndns.win.co.nz/debuild/ubuntu/school/scratch/treecc_0.3.6-0ubuntu1.d{sc,iff.gz} }}} Ok, if you've got those files in a directory now.. {{{ dpkg-source -x treecc_0.3.6-0ubuntu1.dsc }}} In order to unpack them. I'll quickly explain about this after I remember to tell you to rename a file :) {{{ mv treecc-0.3.6.tar.gz treecc_0.3.6.orig.tar.gz }}} before you try the '''dpkg-source''' step :) A debian (or ubuntu) source package is usually made up of upstream source and changes that we have made. Those changes include packaging metadata which say how the package should be built, scripts to run on installation, etc. We keep the upstream tarball as-is, and the debian changes are kept as a single compressed patch against the upstream source. (Please feel free to send myself or '''\sh''', '''siretart''' a msg if you want to interrupt at some point.) The '''treecc_0.3.6-0ubuntu1.dsc''' file that was downloaded has the tarball and the diff.gz listed in it, along with the md5sums of them. '''dpkg-source''' uses this to apply the patch. Now if we go into the '''treecc-0.3.6/''' directory that was created, we see a fairly normal unpacked tarball, except that we now have a '''debian/''' directory which contains the relevant packaging info. In the '''debian/''' dir there are 4 main files I want to talk about now. The '''changelog''' file has a list of the changes made, both fixes we've made to upstream source and packaging changes. As I have just done a single revision, I have 1 changelog entry which is not very informative :) The line: {{{ treecc (0.3.6-0ubuntu1) dapper; urgency=low }}} has some important info. It has the version number, as ''upstream-debianrevision''. It has the distribution (dapper), and the urgency field, which is not used in ubuntu, as far as I'm aware. For ubuntu, we number new packages, or new upstream versions with a debian revision of '''-0ubuntu''X''''', to avoid clashes with debian in the future. A fix to a package 1.2.3-4 would be '''1.2.3-4ubuntu1''', BTW. After this line, changes are listed, in the format shown and the changelog entry is closed off by the name, email and date. Now, onto '''debian/control'''. Quoting from the debian policy manual.. ''The debian/control file contains the most vital (and version-independent) information about the source package and about the binary packages it creates.'' ''The first paragraph of the control file contains information about the source package in general. The subsequent sets each describe a binary package that the source tree builds.'' In this case, we build 1 binary package, so there's only 2 paragraphs of info (source+binary). '''dh_make''' would kindly fill this out for you from a template. (BTW we accept questions at any time ;) ) Debian policy is a very good reference for what we need, but it is not intended as a HOWTO. ||\sh||and we are answering them directly...|| http://www.us.debian.org/doc/debian-policy/ch-controlfields.html#s-sourcecontrolfiles has the relevant info about what fields are required. A question about finding what to put in the '''Build-Depends''' field - much of that is done by reading what the package needs in its README, configure output, etc. ||SWAT||what does the '''${shlibs:Depends}''' stand for?|| '''ajmitch will work through the fields & explain them :)''' I'll go through the source package fields first.. & I'll do a bit of quoting from the policy document to explain. :) * __ Source: __ This field identifies the source package name. Probably one of the easiest for us to fill in. * __ Section: __ This is the debian section that the package comes under. There's a list in debian policy, or we'd look at a similar package to see what section it is in. * __ Priority: __ is used to sort the archive into varying priorities, such as required, standard, optional. It's mainly used in debian & in main, as most of universe would be optional or extra. * __ Maintainer: __ This is the name & email address of the person who created or looks after the package. We generally don't change this in ubuntu if we're making fixes to a debian package but a new package, we set our name & email here. * __ Build-Depends: __ came up earlier - it's a list of packages that are required to build this from source. the exception is build-essential, which is assumed to be installed when building any package. For this package, I read the README file & checked what configure needed in order to build properly. * __ Standards-Version: __ This is the version of debian policy that the package conforms to - there's a list of changes, but it's expected that we follow the latest. the latest is '''3.6.2.1''', but the .1 is for editorial changes, so it can be left off in this file. Now... onto the binary sections :) '''ajmitch will have to try & speed this along''' Feedback is welcome to tell us if this is going too slow, too fast, etc. ||retrix||how do you find out what the latest standards version is?|| aha {{{ dpkg -l debian-policy }}} gives you the version of the installed policy package debian-policy is also available online, from http://www.debian.org/doc/debian-policy/ ok, onwards.. * __ Package: __ this gives the name of the *binary* package that gets created. * The __ Architecture: __ field indicates what arches the packages should build on. It's common to specify all, for a package that is '''architecture-independent''', eg python code or any, for a package that compiles on any architecture. * The __ Depends: __ field gives info about what binary packages this one will depend on when installing - what it needs to run, rather than what it needs to compile.<
><
>'''${shlibs:Depends}''' gets replaced when the package is built to give info about what libraries it needs. * The __ Description: __ field is 2 parts - a short, 1 line description which doesn't need to be a full sentence and a paragraph or so of text about what it is. A question about "'''architecture: any'''" vs. "'''architecture: all'''" Packages that are "'''arch: __all__'''" get built once, and work on all architectures - they are usually data files, python code, shell scripts, etc. "'''arch: __any__'''" gets compiled for any arch - you could list '''i386 amd64 powerpc''' instead to limit it to these archs. '''ajmitch would like to move onto the big debian/rules if there are no objections, since this is taking a little longer than I expected :)''' '''siretart hasn't received any question yet, feel free to ask anything which you think that might be noteworthy''' '''debian/rules''' is the main file that controls how everything gets built - it is a makefile, with specific required make targets. As an example, I have the same package done 3 ways ('''debhelper''', '''cdbs''', without helper programs) to compare. :) 170 lines for this '''debian/rules'''<
>61 lines with '''debhelper'''<
>and 12 with '''cdbs''' You can probably tell that this way is more verbose & more complex :) Again, from debian policy.. ''This file must be an executable makefile, and contains the package-specific recipes for compiling the package and building binary package(s) from the source.'' ''It must start with the line "{{{#!/usr/bin/make -f}}}", so that it can be invoked by saying its name rather than invoking '''make''' explicitly.'' ''Since an interactive '''debian/rules''' script makes it impossible to auto-compile that package and also makes it hard for other people to reproduce the same binary package, all required targets MUST be non-interactive. At a minimum, required targets are the ones called by '''dpkg-buildpackage''', namely, '''clean''', '''binary''', '''binary-arch''', '''binary-indep''', and '''build'''. It also follows that any target that these targets depend on must also be non-interactive.'' Basically, it says that we need these few makefile targets, and they can't ask the builder anything since everything uploaded in ubuntu gets built on a build daemon, unattended. We might as well look at the '''debian/rules''' now. Firstly, I started off by setting up some commonly used variables. The '''DEB_BUILD_OPTIONS''' environment variable can control some aspects of building, often for things like not stripping a binary. Some small macros are defined to check that we are in the right directory & have uid 0. '''fakeroot''' is often used to fake the existence of uid 0 when building, since packages can & should be built as a user. '''DESTDIR''' is important, as it's an option passed to make to tell it where to relocate the installed files to when running make install. So it's used throughout, in the other directory variables. We have listed 2 '''stamp''' rules, which are used to avoid repetition. '''make''' was originally meant for building plain source, so it needs some files to be created to avoid running targets again. * The '''configure-stamp:''' target is what contains the call to configure. In this case, we're not passing many configure options, but some packages do take quite a few, which can be put in here. * The '''build-stamp:''' target has '''configure-stamp''' after it, to indicate that it needs '''configure-stamp''' to run first, before building. It first checks that we're in the right directory, and runs make. The debian rules section is what is required by debian policy. * '''build:''' just depends on '''build-stamp''', and does nothing more - if '''build-stamp''' was run it won't repeat it. * The '''clean:''' target gets rid of everything, and restores things to a (relatively) pristine state. if patches had been applied, they would be unapplied here. stamp files are removed, and make distclean is run to get rid of the build output. * '''binary-indep:''' is another fairly empty rule, which just depends on build. If this package was '''Architecture: all''', we'd be doing a lot more in here. '''ajmitch now has a few lines to give options to install, to avoid typing them over & over later on''' * The '''binary-arch:''' target is now where we do most of the actual packaging of the files which is why I've commented this one the most. :) If there are any questions up till now, please ask We have a question: "How if the '''distclean''' rule from the source '''Makefile''' doesn't clean properly the source? Is there a way to restore it in the '''clean:''' rule?" If '''distclean''' doesn't properly clean things, then often we resort to just using '''rm''' in the '''clean:''' rule. Upstream authors often don't have perfect makefiles that clean things. :) If '''distclean''' fails (eg the makefile doesn't exist yet), then it will still continue. ||Laser''''''Jock||do you have to have the stamps?|| The stamp files aren't needed, but they're useful to avoid running things again, especially if it was a long compile & you just want to test out packaging changes. Then you could run the build target once & the '''binary-arch''' target multiple times to get things right. It's not as useful running in '''pbuilder''', when we're always doing a clean build. Any other questions about that? ok, will move on down the file :) First step is installing what we've built, into the directory '''DESTDIR'''. So you end up with a directory structure like '''debian/treecc/usr/bin''', for example. '''ajmitch chose to put '''make check''' in there, to run the program test suite. No point packaging up a broken program, so we check it where we can.''' Now, what we don't see with using '''debhelper''' or '''cdbs''' is the '''DEBIAN''' directory which is '''debian/treecc/DEBIAN'''. The '''control''', '''changelog''', '''maintainer''' scripts go in here - '''debhelper''' hides this from us nicely. This next section about installing docs is handled by 1 '''debhelper''' program, in this case we just copy the files into the right place. I'll get to the '''debian/copyright''' file after this, it's also rather important. Now, the package scripts - for this package we have a script to run after installation, eg 'Setting up treecc (0.3.6-0ubuntu1)'. The script I have is very simple, all it does is register the info documentation. You can look at it in '''debian/postinst''' if you want to - I'll try not to spend too much time on this right now. :) The '''prerm''' script is subsequently called when removing or upgrading the package. The URL in the comments has details of how these interact. After this, we compress the documentation - I didn't compress the '''README''' or '''NEWS''' files, although I could have here. A common thing is to also have a '''README.Debian''' file, which has information specific to the package on debian/ubuntu. I was also lazy & didn't install the other documentation & example files, apart from the manual (in info format). Questions came up earlier about '''${shlibs:Depends}''' in '''debian/control'''. '''dpkg-shlibdeps''' works out what libraries are needed by the binaries, via a nice system of magic. Library packages have shlibs files to indicate the binary package version required of the library when built against the library headers. These are used & put into the '''debian/substvars''' file by '''dpkg-shlibdeps'''. There can be a number of lines in '''substvars''', but the most common one you'll ever see or use will be '''${shlibs:Depends}'''. Another common one used with '''debhelper''' is '''${misc:Depends}'''. '''dpkg-gencontrol''' will take the file from '''debian/control''', and generates a control file suitable for the binary package in '''debian/treecc/DEBIAN/control'''. It will take those '''substvars''' & do the replacements needed - more info on that is available in the '''dpkg-gencontrol''' manpage, in the VARIABLE SUBSTITUTION section. The next part is commonly done by '''dh_md5sums''' when using '''debhelper''' - here we have to generate md5sums manually for each file that will get installed. This is mainly used by '''debsums''' (if you choose to use it) to track modified files. Now, the final part in this target is '''dpkg-deb''', which does the part of taking the files in '''DESTDIR''', and generating a binary package. It takes the files in '''DEBIAN''' & puts them in the '''control.tar.gz''', and the rest from '''debian/treecc''' into '''data.tar.gz''', and packs it into the '''ar''' archive. So, by the time we get here, we have a binary package built. :) any questions? Might as well allow some public questions for now. :) '''ajmitch hopes it hasn't been too confusing or too slow for people.''' '''ajmitch sets mode: -m ''' ---- ||siretart||public question time!|| '''ajmitch still wants to briefly talk about the copyright file after this<
>ajmitch will step out for a couple of minutes, ask questions while I'm afk :)''' ||retrix||Whats the next step after building a package, i.e. how do we get it into universe?|| ||Kyral||Submit to REVU for review|| ||raphink||retrix: for that, have a look at the REVU website: http://revu.tauware.de/|| ||nmsa||What I don't see are cdbs and debhelper. Will this follow?|| ||raphink||nmsa: maybe in the next lecture ;)|| ||\sh|| nmsa: this will come later :) we will have the cdbs maintainer of debian talking to us...|| ||Kyral||nmsa: This was meant to show you what Debhelper and CDBS do for you|| ||SWAT||aha :)|| ||SWAT||they do quite a freaking lot for us|| ||nmsa||Kyral: in comparison to make, right?|| '''sladen has been keeping a rough-and-ready summary at: http://www.paul.sladen.org/ubuntu/tutorial/01/commands.txt''' please note, I haven't quite finished yet ;) ||siretart||are there any questions left?|| ||siretart||ajmitch wants to loose some words on debian/copyright|| as well as getting the thing built ;) ||Kyral||If no one has an idea for the next lesson I could do one on PBuilders|| ||\sh||Kyral: lets discuss this later...I'll have to make all the schedules :)|| ||Kyral||I have to go up to the Linux Lab|| ||siretart||what does the auditorium think about +m?|| ||raphink||siretart: not sure it's necessary|| Ok, onto debian/copyright then You might note that the date is fairly old - that was when I first built this package with dh_make. The copyright statement in this package is a little messy. As it has the standard 3 paragraphs of GPL, with a reference to the full text and then it has some further explanation from upstream about the licensing of output of this program. If the program has certain exceptions like this, or linking exceptions, then you need to put them in here. This should ideally have a copyright (c) xxxx-yyyy Southern Storm Software in it Right, I think that's most of what we need for the debian/ files. If you want, you could build the package now with '''dpkg-build-package -rfakeroot''' you see what happens in debian/rules then all I can say is that it 'works for me'. ;) ||stephank||what's the difference when I use debuild?|| shorter to type ;) '''debuild''' runs '''dpkg-buildpackage''', then '''lintian'''/'''linda'''. Then it will sign the '''.changes''' and/or '''.dsc''' file created. I always prefer to prepare a source package for upload with '''debuild -S''' (and other options where needed) since everything needs to be signed before upload. ||siretart||stephank: '''debuild''' is basically a wrapper for '''dpkg-buildpackage'''. It is part of the package '''devscripts''', and can be nicely configured in '''~/.devscripts'''. I've set in '''~/.devscripts''' that I do not want to sign my packages with '''debuild''', but manually with '''debsign'''. (I consider signing packages as part of the upload process, not the build process).|| '''ajmitch does it the other way, signing unless -us -uc is passed, but that's because I just have the signing habit.''' ||stephank||I see, thanks. :)|| This won't be uploaded to the archive, 0.3.6-2 is already in there. :) any other questions? Perhaps questions about the '''debhelper''' or '''cdbs''' options? if you want to compare rules files :) * http://ubuntu.pastebin.com/458503 for '''debhelper-less''' * http://ubuntu.pastebin.com/458488 for '''debhelper''' * http://ubuntu.pastebin.com/458490 for '''cdbs''' Using debhelper is often simple, but it doesn't require that you know what is going on. ||akurashy||What's cdbs?|| '''cdbs''' is even simpler for a standard package. '''cdbs''' = '''c'''ommon '''d'''ebian '''b'''uild '''s'''ystem. It abstracts away nearly everything. So '''debhelper''', etc are even hidden from the packager. ||sistpoty||it's a set of make-rules that you can include in debian/rules|| ||Laser''''''Jock||so is there some kind of "rule of thumb" for when to use the different systems?|| Use what feels most appropriate :) '''cdbs''' has its limitations, if you find yourself having to hack round things then you know that it might not be best ||nmsa||what are *.mk files?|| they're just makefiles ||Laser''''''Jock||so would '''cdbs''' have a lot of overhead for a small packages?|| ||sistpoty||Laser''''''Jock: yep|| well, not a lot of overhead ||siretart||Laser''''''Jock: the good thing about '''debhelper''' is, that if it breaks, you know excatly where it broke|| '''treecc''' is a relatively small package, and it uses '''cdbs''' ||siretart||Laser''''''Jock: with '''cdbs''', you don't get that information easily|| '''cdbs''' requires some serious make-fu if you need to move outside the box ||siretart||Laser''''''Jock: so for small packages, '''cdbs''' might be okay. For big and complicated ones, '''debhelper''' might be easier for maintenance|| It scares me, and I'm sure it scares its author. :) But 90% of the time it's great for what I want. ||Laser''''''Jock||so maybe for a package that doesn't require any funky stuff '''cdbs''' would be good but if you need to do quite a bit of tweaking then '''debhelper''' would be better?|| yeah, or another build system ||siretart||ajmitch: please dont recommend yada *g* haha|| ||Laser''''''Jock||'''cdbs''' just scares my right now because it is too magical ;-)|| ||slomo||Laser''''''Jock: but it safes you much time for standard packages ;)|| certainly I managed to get a package done in around 5 minutes one time with '''cdbs''' :) and uploaded about 30 minutes later after some MOTUs checked it :) ||slomo||ajmitch: hehe dito, but only because copyright was easy for this one ;)|| ok, 2 hours, I guess that's probably about long enough unless there's other questions :) ||stephank||Perhaps a bit unrelated, but why is '''dpatch''' bad? (read that somewhere) Is it replaced by '''debhelper''' or '''cdbs'''?|| stephank: '''dpatch''' is not bad, as such and it's a patch system that is used along with debhelper/cdbs. It's 'bad' in that we shouldn't be adding '''dpatch''' to packages that don't use it in debian since that just introduces intrusive changes ||stephank||Okay. It has an odd patch format, IMO. :)|| Odd? '''dpatch''' has a fairly standard patch format, just with a header. ||minghua|| another reason people consider '''dpatch''' "bad" is because '''dpatch''' doesn't have a standard debian/rules target to apply all needed patches or simple-patchsys with '''cdbs'''|| ||Laser''''''Jock||are there other patch systems out there?|| yes, there are several others ||tseng||Laser''''''Jock: none of them are suitable to add to packages that dont use them.|| ||Laser''''''Jock|| tseng: I agree I just wonder about what to use when making new packages I tend towards dpatch for my packages.|| ||tseng||'''dpatch''' or '''simple-patchsys''' for '''cdbs'''. both are plain patches. '''dpatch''' with a header|| ||minghua||This is a main complain from security people because they need to look at and patch other people's packages quite often yes, it can be annoying as do MOTUs :)|| ||siretart||excatly|| ||tseng||the complaints are off topic, its enough to say "dont add a patch system to someone elses package"|| ||minghua|| If the maintainer uses '''dpatch''' and has some obscure debian/rules, it takes quite some time to figure out the correct way to get the source tree to patch against|| ||sistpoty||debian source packages are a kind of patch system in itself, since the '''.diff.gz''' is basically a patch.|| ||\sh||Ok...any other questions can be raised on ubuntu-motu mailing list|| ---- (Thanks and kudos followed...) ---- [[CategoryArchive]]