Ubuntu Wiki

Ubuntu Open Week - Patching Packages - Tue, Nov 28, 2006

see also Thursday Session.

08:00   pitti   WELCOME TO MY CLASS
08:01   pitti   I have a rather technical presentation now, for everyone who wants to modify anything in Ubuntu
08:01   pitti   just as a quick overview, who is here to learn about patching?
08:02   pitti   great!
08:02   pitti   ok, time's limited, let's start
08:02   pitti   if anyone has any question, or I'm totally uncomprehensible (sorry, I'm German), please do not hesitate to interrupt and ask *immediately*
08:02   pitti   Also, don't bother trying to take notes, we'll sort that out at the end. You can fully concentrate on the discussion and examples.
08:02   pitti   Let's begin with a little bit of history:
08:02   pitti   == Why use separate patches ==
08:02   pitti   In earlier times, people just applied patches inline (i. e. directly in the source code tree). However, this makes it very hard to extract patches later to modify them, send them upstream, etc. Also this means that new upstream versions are a pain, since they generate a lot of rejections when applying the package diff.gz to them.
08:02   pitti   With split-out patches it is much easier to send them upstream, keep track of them, develop them, etc., since you always see which changes belong together.
08:02   pitti   The ideal state is an unmodified tarball from upstream, plus clean and separate patches, plus the packaging bits in debian/. That means that lsdiff <sourcepackage>.diff.gz only contains debian/.
08:02   pitti   The first attempts to split-out patches were pretty trivial: storing patches in debian/patches/, and adding some patch/patch -R snippets to debian/rules. This worked for small patches, but provided no tools for editing these patches, updating them for new upstream versions, etc.
08:02   pitti   Thus several standard patch systems were created which are easy to deploy and provide tools for patch juggling and editing.
08:04   pitti   What I would like to do now is to introduce the most common patch systems and show some hands-on demo how to add a new patch and how to edit one. For this, I will point at a source package from the current dapper archive, quickly explain the patch system, and show how to apply some (braindead) modifications to it. I recommend you to do the same steps in a terminal, so that you get a feeling for the process and can immediately ask questions.
08:04   pitti   everyone you fine with this approach?
08:04   pitti   erm, of course I mean 'from the edgy archive', not dapper
08:04   pitti   yeah, I don't want it to be entirely theoretical :)
08:05   pitti   If you want to try the stuff yourself, please do the following commands (on edgy) as preparation:
08:05   pitti     sudo apt-get install dpatch cdbs quilt patchutils
08:05   pitti     apt-get source cron udev pmount gnome-volume-manager ed xterm
08:05   pitti     wget http://people.ubuntu.com/~pitti/scripts/dsrc-new-patch
08:05   pitti     chmod 755 dsrc-new-patch
08:05   pitti   I deliberately picked the smallest packages I could find
08:08   pitti   ok, everyone set with the intro and getting source packages?
08:09   pitti   nevermind, for the first bit we don't yet need the packages
08:09   pitti   == cron: inline patches ==
08:09   pitti   No patch system at all, nothing much to say about this.  You directly edit the files in the source tree. This is convenient for a simple and quick change, but will bite back for new upstream versions (see above) and is inconvenient for submitting patches upstream, or reviewing for merges.
08:09   pitti   if you do 'lsdiff <package>.diff.gz' and you see changes which are not in debian/, then you probably have such a package
08:09   pitti   (some KDE packages have autoconf stuff directly in the diff.gz, but that is ok)
08:09   pitti   so, I think I do not need to say anything else about cron, unless someone has a question
08:10   pitti   ok, no question here
08:10   pitti   so, people, get a grip, now it gets heavy
08:11   pitti   == udev: separate patches, but no patch system ==
08:11   pitti   This case is the most complicated one since you have to do all the hard work manually. In order to make you understand what a patch system does, and to give you a fallback method that will *always* work with any patch system, I handle this first.
08:11   pitti   The good news is that you will seldomly be required to actually do this procedure, I'll come to this again.
08:11   pitti   The general approach is:
08:11   pitti   1. copy the clean source tree to a temporary directory /tmp/old
08:11   pitti   2. apply all patches up to the one you want to edit; if you want to create a new patch, apply all existing ones (this is necessary since in general patches depend on previous patches)
08:11   pitti   if you want, you can use debian/rules for this: remove the patches that come *after* the one you want to edit, and call 'debian/rules patch'. The actual name for the patch target varies, I have seen the following ones so far: patch setup apply-patches unpack patch-stamp. You have to look in debian/rules how it is called.
08:11   pitti   3. copy the whole source tree again: cp -a /tmp/old /tmp/new
08:11   pitti   4. go into /tmp/new, do your modifications
08:11   pitti   5. go back into /tmp generate the patch with
08:11   pitti     diff -Nurp old new > mypatchname.patch
08:11   pitti   and move the patch to .../debian/patches/mypatchname.patch
08:11   pitti   ^ that's the theory, example will follow shortly (hands-on training)

[jonh_wendell] pitti, i do a lsdiff <package>.diff and got nothing

• that won't work in that form - lsdiff -z package.diff.gz. -z is for 'accept gzipped files', source packages have a diff.gz, not a .diff

08:12   pitti   these steps are the general method how any patch is handled
08:14   pitti   in general we want the following diff options:
08:14   pitti   -N -> include new files
08:14   pitti   -u -> unified patches (context diffs are ugly)
08:14   pitti   -r -> recursive
08:14   pitti   -p -> bonus, you can see the name of the affected function in the patch
08:14   pitti   does anyone have a question about the principle method?
08:14   pitti   (I know it's heavy, sorry, that's live; promised, it'll get easier)

[jonh_wendell] pitti, about cron, are you showing a wrong way to package?

• well, it's not exactly 'wrong' as long as the package is not heavily modified, or Debian/Ubuntu is essentially the upstream, it's fine, but having all patches in a single big lump makes it next to impossible to send stuff upstream, or tell which bit belongs to what. also, the tendency is that packages which have 'always' been there, have more 'archaic' build systems and no patch system where as most recently packaged stuff uses a standard packaging system with a standard patch system, now that they are invented

[mwicki] does anybody know how to install a debianpaket whitout root-permission, is it possible to install a debian-paket in a directory??

• [Jon] you can unpack it using dpkg-deb -x <package> <directory> as any user who can write to <directory>. mwicki: be warned, though, that doesn't run the pre or post install scripts do you need to do that for patching? it most probably won't work. if it's a simple program then you could then run the binaries in it

08:17   pitti   ok, let's go back to patching udev - an EXAMPLE! \o/
08:17   pitti   open a shell, ready your fingers :)
08:17   pitti   udev example 1, let's create a new patch 90_penguins.patch:
08:17   pitti     cd /whereever/you/unpacked/the/source/udev-093
08:17   pitti     cp -a . /tmp/old
08:17   pitti     pushd /tmp/old
08:17   pitti     debian/rules patch
08:17   pitti     cp -a . /tmp/new; cd ../new
08:17   pitti   that's the preparatory part, now we have a directory we can hack in
08:18   pitti   let's do a braindead modification
08:18   pitti     sed -i 's/Linux/Penguin/g' README
08:18   pitti   ^ of course you can also use a normal editor
08:18   pitti   I just use sed so that you can copy&paste the commands without any interactive stuff
08:19   pitti   and now we create a patch between the reference and our new tree
08:19   pitti     cd ..
08:19   pitti     diff -Nurp old new > 90_penguins.patch
08:19   pitti     popd
08:19   pitti     rm -rf /tmp/old /tmp/new
08:19   pitti     mv /tmp/90_penguins.patch debian/patches
08:19   pitti   (Ignore the 'recursive directory loop' warnings)

[jonh_wendell] what does pushd do?

• 'pushd foo' is like 'cd foo', but remembers the last dir and pushes it onto a stack. if you then do 'popd', it will return to the directory you have been in when you did 'pushd'

  [StaCk] isn't it like cd - ? no, that will go only to the previous dir. you can stack multiple pushd's on top of each other, and allow 'cd' in between

08:21   pitti   Now take a look at your shiny new debian/patches/90_penguins.patch.
08:21   pitti   *uff* :)
08:21   pitti   after that, if you do 'debian/rules patch', you'll see that the patch applies cleanly
08:21   pitti   
08:22   pitti   so, obviously that's not the end of the wisdom, but if you do these steps a couple of times, you should get a feeling for how to create the most complicated patch conceivable
08:22   pitti   so this procedure is the life safer if anything else fails
08:22   pitti   questions so far?

[pschulz01] pitti: cd new?

• 'cp -a . /tmp/new; cd ../new' We applied all previous patches in /tmp/old, then copied /tmp/old to /tmp/new, and hacked our stuff in /tmp/new

[dieterd] result is then a new cd with included patches?

• the result after all of these steps is a single new file debian/patches/90_penguins.patch. the commands above removed the temporary /tmp/old and /tmp/new trees

[pschulz01] Need to copy 90_penguins.patch to new/debian/patches

• no, it shouldn't go to new. it should go to /whereever/you/unpacked/the/source/udev-093 (the original source dir, not the temporary working copies in /tmp)

08:25   pitti   Pretty much work, isn't it? Since this happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose.
08:25   pitti   that's the script you downloaded as part of the preparations
08:26   pitti   so, please remove debian/patches/90_penguins.patch again
08:26   pitti   yes, I know it was hard to do, but for the sake of demonstration :)
08:26   pitti   so, with Pitti's Hackish Script (tm):
08:26   pitti     dsrc-new-patch 90_penguins.patch
08:26   pitti     sed -i 's/Linux/Penguin/g' README
08:26   pitti     <press Control-D to leave the subshell>
08:27   pitti   that looks slightly better, doesn't it?
08:27   pitti   but I had to torture you with the close-to-the-metal method for the sake of understanding.

[at2000] in which dir to run the script?

• in the root of the unpacked source tree to conform with my previous example, /whereever/you/unpacked/the/source/udev-093

  [at2000] then it should be "../dsrc-new-patch"

  • oh, right, you have to specify the path to the scrpit. I have it in my ~/bin, conveniently

[jonh_wendell] is this script the same of cdb... showed yesterday?

• no, not at all. I'll come to that

[pschulz01] Hmm.. where do I find 'debclean'?

• sudo apt-get install devscripts. sorry; forgot too mention that

[jonh_wendell] before run dsrc-new-patch 90_penguins.patch we have to clean dir, right?

• if you just unpacked the tree, it shuoldn't be necessary

08:31   pitti   ok, we need to go on
08:31   pitti   let's stay at udev a bit and do the second example
08:31   pitti   I promise, this will be the last complicated issue in this lesson
08:31   pitti   but it might teach you to never ever do a package without a proper patch system :)
08:31   pitti   dsrc-new-patch is currently too dumb to edit existing patches, or to put patches somewhere else than the top of the patch stack. If you need this, then you need to do the manual approach. Assume we want to edit 50-result-whitespace.patch.

[jonh_wendell] as i've run debian/rules patch with 90-penguim, i have to clean it

• ah, right

08:32   pitti   This already existing patch 50-result-whitespace.patch does not depend on the previous patches, so it's a bit easier that way, too:
08:32   pitti     cd /whereever/you/unpacked/the/source/udev-093
08:32   pitti     cp -a . /tmp/old
08:32   pitti     pushd /tmp/old
08:32   pitti     cp -a . /tmp/new; cd ../new
08:32   pitti   nothing new so far
08:32   pitti   but note that we did not apply any patch (since we assume that 50-result-whitespace.patch does not depend on its predecessors)
08:33   pitti     patch -Nlp1 < debian/patches/50-result-whitespace.patch
08:33   pitti   above will apply the current version of the patch, so that you can edit it
08:33   pitti   now we have the *original* change in /tmp/new, which we now want to modify
08:33   pitti   so, let's do another braindead change:
08:33   pitti     sed -i '1 s/$/***** HELLO WORLD ****/' udev_selinux.c
08:34   pitti   or any other change you quickly want to do in your favourite editor, of course
08:34   pitti   so, again we now have the original version in /tmp/old, and our old and new patch in /tmp/new
08:34   pitti   so let's do the patch:
08:34   pitti     cd ..
08:34   pitti     diff -Nurp old new > 50-result-whitespace.patch
08:34   pitti     popd
08:34   pitti     mv /tmp/50-result-whitespace.patch debian/patches
08:34   pitti     rm -rf /tmp/old /tmp/new
08:34   pitti   Now debian/patches/50-result-whitespace.patch contains both the original change and our modification of the first line of the source file.
08:35   pitti   but promised, from now on it will get really easy :)
08:35   pitti   any questions? is the issue of patch dependencies clear to everyone?

[jonh_wendell] how do i know if a patch depends on others or not?

• in general you should assume that each patch depends on the previous one. they *don't* if they patch a file that hasn't been patched before or they patch it on a completely different position. e.g., if debian/patches/1_foo.patch changes foo.c, line 5 and debian/patches/2_bar.patch changes foo.c at line 6 then patch 2 definitively depends on patch 1. a (non-solid) rule of thumb is: if you can apply the patch alone on the unmodified source tree, it's independent

[at2000] is unified patch format better eliminating patch dependency?

• no, not at all. unidiff vs. context diff is purely a matter of readability. patch dependency is an inherent property of patches modifying the same thing not a question of a particular patch format

08:38   pitti   Since this is so hideously complicated, patch systems were invented to aid you with that.
08:38   pitti   Let's look at the most popular ones now (they are sufficient to allow you to patch about 90% of the archive's source packages; for the rest you have to resort to the manual approach above).
08:39   pitti   == pmount: cdbs with simple-patchsys ==
08:39   pitti   cdbs' simple-patchsys.mk module matches its name, it has no bells and whistles whatsoever. However, it is pretty popular since it is sufficient for most tasks, and long ago I wrote a script 'cdbs-edit-patch' which most people can live with pretty well. This script is contained in the normal cdbs package.
08:39   pitti   You just supply the name of a patch to the script, and depending on whether it already exists or not, it will create a new patch or edit an existing one.
08:39   pitti   simple-patchsys is the cdbs module that 90% of all cdbs-using source packages will use
08:39   pitti   BTW, "cdbs-edit-patch" is slightly misleading, since it actually only applies to simple-patchsys.mk. You can also use other cdbs patch system plugins, such as dpatch or quilt.
08:40   pitti   everyone please look in debian/patches, debian/rules of pmount to get a feeling how it looks like
08:40   pitti   so, let's mess up pmount a bit
08:40   pitti   and add a new patch
08:40   pitti     cd /whereever/you/unpacked/the/source/pmount-0.9.13
08:40   pitti     cdbs-edit-patch 03-simple-readme.patch
08:40   pitti     echo 'This should document pmount' > README
08:40   pitti     <press Control-D to leave the subshell>
08:40   pitti   easy, isn't it?
08:41   pitti   check debian/patches/03-simple-readme.patch for the result
08:41   pitti   this will take care of applying all patches that need to be applied, can change patches in the middle of the stack, and also create new ones
08:41   pitti   Editing an already existing patch works exactly the same way, so I won't give a demo
08:42   pitti   unfortunately it only really works for packages that use cdbs
08:43   pitti   which a good part of Gnome-related packages do

[jonh_wendell] how do i know if a package uses cdbs?

• debian/rules includes files from /usr/share/cdbs and build-depends on cdbs

[pschulz01] What does cdbs stand for?

• common debian build system. it's a very modular abstraction of boilerplate build rules, patch systems, etc.

[at2000] when I cdbs-edit-patch the same diff, it results in 2 diffs concatenated in the same file?

• if you edit two different files, yes. in general, it will merge the old and new patch

[jonh_wendell] when we make a patch, should we add entries in debian/changelog file too?

• if you intend to publish this, yes, by all means

08:46   pitti   == ed: dpatch ==
08:46   pitti   dpatch is a pretty robust and proven patch system which also ships a script 'dpatch-edit-patch'
08:46   pitti   The two most important things you should be aware of:
08:46   pitti   * dpatch does not apply debian/patches/*, but instead applies all patches mentioned in debian/patches/00list, in the mentioned order. That means that you do not have to rely on asciibetical ordering of the patches and can easily disable patches, but you have to make sure to not forget to update 00list if you add a new patch.
08:46   pitti   (forgetting to update 00list is a common cause of followup uploads)
08:46   pitti   * dpatch patches are actually scripts that are executed, not just patches fed to 'patch'. That means you can also do fancy things like calling autoconf or using sed in a dpatch if you want.
08:47   pitti   using dpatch for non-native patches is rare, and normally you do not need to worry about how a .dpatch file looks like
08:47   pitti   but I think it's important to mention it
08:47   pitti   so if you ever want to replace *all* instances of Debian with Ubuntu in all files, write a dpatch with a small shell script that uses sed
08:47   pitti   instead of doing a 300 KB static patch which won't apply to the next version anyway
08:47   pitti   The manpage is very good and has examples, too, so I will only give an example here:
08:47   pitti   This will edit an already existing patch and take care that all previous patches are applied in order:
08:48   pitti     cd /whereever/you/unpacked/the/source/ed-0.2
08:48   pitti     dpatch-edit-patch 05_ed.1-warning-fix
08:48   pitti     <edit stuff, press Ctrl+D>
08:48   pitti   so that's exactly like cdbs-edit-patch
08:48   pitti   well, dpatch-edit-patch even has shell completion (!)
08:48   pitti   s/shell/tab/
08:48   pitti   ok, now we edited a patch, that's pretty easy, right?
08:49   pitti   now let's create a new one; this is different from cdbs-e-p, due to the explicit '00list' file
08:49   pitti     dpatch-edit-patch foo.dpatch 06_testsuite-Makefile.dpatch
08:49   pitti     <edit stuff, press Ctrl+D>
08:49   pitti     echo foo.dpatch >> debian/patches/00list
08:49   pitti   ^ this will append 'foo.dpatch' to debian/patches/00list
08:50   pitti   This will create a new patch foo.dpatch relative to the already existing 06_testsuite-Makefile.dpatch. If your patch is very confined and does not depend on other patches, you can leave out the second argument.
08:50   pitti   alright?
08:50   pitti   there is actually yet another patch system I wanted to introduce (quilt), but I'll skip this for time reasons
08:51   Seveas  you could continue after simon finished talking bugs
08:51   pitti   maybe we can get to that on Thursday, or I'll explain it separately
08:51   pitti   and there is existing documentation anyway
08:51   pitti   so, just some more theoretic words, which you can use to calm down your fingers :)
08:51   pitti   === A glimpse into the future ===
08:52   pitti   As you saw, Debian source packages do not have any requirements wrt. structure, patch systems, etc., other source package systems like SRPM are much stricter wrt that. This of course means more flexibility, but also much more learning overhead.
08:52   pitti   As a member of the security team I can tell tales of the pain of a gazillion different source package layouts... :)
08:52   pitti   Therefore some clever people sat together the other day to propose a new design which would both give us a new and unified source package and patch system that uses bzr (with a quilt-like workflow). This would also integrate packages and patches much better into Launchpad and revision control in general.
08:52   pitti   Please take a look at https://wiki.ubuntu.com/NoMoreSourcePackages if you are interested in this.
08:52   pitti   ah, and of course:
08:53   pitti   === Notes ===
08:53   pitti   (apart from the obligatory channel log from fabbione)
08:53   pitti   There is also a wiki page https://wiki.ubuntu.com/MOTU/School/PatchingSources which provides most of above information in a more convenient format. However, it might be slightly out of date (it's from dapper times). Feel free to update the page and and add missing bits.

[jonh_wendell] we should add manually foo.dpatch in 00list, right?

• right. that's what I did with "echo foo.dpatch >> debian/patches/00list". This will append 'foo.dpatch' to debian/patches/00list

[at2000] what are pros and cons of quilt?

• hm, matter of taste, mainly. the patching workflow is not really different

[at2000] so why not just stick to cdbs, what most packages use?

• we cannot prescribe Debian to change source packages to our wishes. also, cdbs is not the unanimous solution for Ubuntu devs either. some people really hate it

[ajmitch] has there been any more progress on the NoMoreSourcePackages spec?

• bzr-wise, yes, but there's no LP support yet and it won't come in the near future, there are much more pressing TODOs for soyuz ATM

08:56   pitti   I know it was hard stuff, I hope I didn't kill too many listeners
08:56   pitti   any feedback for me?
08:57   pitti   (but patching source packages is not easy...)

[jonh_wendell] any talk about debdiff? is it a diff of a diff?

• 'debdiff' is a tool that gives you a diff between two complete source packages. so, if you debdiff two source packages which include patches, then yes, you'll get diffs of diffs

09:00   pitti   would it have been better to first explain the tools and then do the manual approach?
=== pitti will try this on Thursday