Ubuntu Open week - Patching Packages - Martin Pitt - Thu, Apr 26, 2007
see also Tuesday Session.
(01:01:08 PM) pitti: welcome to my hands-on training about patching packages, an inevitable tool for new developers (01:01:23 PM) pitti: as a warning, this will be the very same talk that I did on Tuesday (01:01:32 PM) pitti: so if you were in that, this won't be particularly interesting (01:01:39 PM) pitti: if anyone has any question, or I'm totally uncomprehensible (sorry for my English, I'm German), please do not hesitate to interrupt and ask *immediately* (01:01:48 PM) pitti: Also, don't bother trying to take notes, we'll sort that out at the end. You can fully concentrate on the discussion and examples. (01:02:04 PM) pitti: who here would learn about patching packages? (quick audience straw-poll) ... (01:02:52 PM) pitti: that sounds more than manageable :) (01:03:01 PM) pitti: Let's begin with a little bit of history: (01:03:07 PM) pitti: == Why use separate patches == (01:03:12 PM) pitti: In earlier times, people just applied patches inline (i. e. directly in the source code tree). However, this makes it very hard to extract patches later to modify them, send them upstream, etc. Also this means that new upstream versions are a pain, since they generate a lot of rejections when applying the package diff.gz to them. (01:03:24 PM) pitti: With split-out patches it is much easier to send them upstream, keep track of them, develop them, etc., since you always see which changes belong together. (01:03:33 PM) pitti: The ideal state is an unmodified tarball from upstream, plus clean and separate patches, plus the packaging bits in debian/. That means that lsdiff -z <sourcepackage>.diff.gz only contains debian/. (01:03:51 PM) pitti: oh, btw, I assume that you already know what a source package is and how it looks like in general (01:04:04 PM) pitti: The first attempts to split-out patches were pretty trivial: storing patches in debian/patches/, and adding some patch/patch -R snippets to debian/rules. This worked for small patches, but provided no tools for editing these patches, updating them for new upstream versions, etc. (01:04:23 PM) pitti: (we will see why that is painful) (01:04:28 PM) pitti: Thus several standard patch systems were created which are easy to deploy and provide tools for patch juggling and editing.
<bullgard4> QUESTION: Will you speak about patching a module for everybodys use or for one'own use?
- that doesn't matter much. of course packages that will be uploaded need to be more strict and cleaner, but it's still useful if you personally do a patch and just want to send it to someone, or keep it around for a while
(01:05:39 PM) pitti: What I would like to do now is to introduce the most common patch systems and show some hands-on demo how to add a new patch and how to edit one. For this, I will point at a source package from the current feisty archive, quickly explain the patch system, and show how to apply some (braindead) modifications to it. I recommend you to do the same steps in a terminal, so that you get a feeling for the process and can immediately ask questions. (01:05:52 PM) pitti: is everyone fine with this approach? ... (01:06:49 PM) pitti: If you want to try the stuff yourself, please do the following commands (on feisty) as preparation: (01:06:55 PM) pitti: sudo apt-get install dpatch cdbs quilt patchutils devscripts (01:06:55 PM) pitti: apt-get source cron udev pmount gnome-volume-manager ed xterm (01:06:55 PM) pitti: wget http://people.ubuntu.com/~pitti/scripts/dsrc-new-patch (01:06:55 PM) pitti: chmod 755 dsrc-new-patch (01:07:05 PM) pitti: I deliberately picked the smallest packages I could find (01:07:09 PM) ***pitti waits a bit for people to do the preparations; any questions so far? ... (01:09:15 PM) pitti: == cron: inline patches == (01:09:22 PM) pitti: No patch system at all, nothing much to say about this. You directly edit the files in the source tree. This is convenient for a simple and quick change, but will bite back for new upstream versions (see above) and is inconvenient for submitting patches upstream, or reviewing for merges. (01:09:49 PM) pitti: just because all patches you apply will end up as a giant big mess in the diff.gz (01:09:56 PM) pitti: if you do 'lsdiff -z <package>.diff.gz' and you see changes which are not in debian/, then you probably have such a package (01:10:07 PM) pitti: so, I think I do not need to say anything else about cron, unless someone has a question (01:10:54 PM) pitti: == udev: separate patches, but no standard patch system == (01:10:59 PM) pitti: This case is the most complicated one since you have to do all the hard work manually. In order to make you understand what a patch system does, and to give you a fallback method that will *always* work with any patch system, I handle this first. (01:11:08 PM) pitti: The good news is that you will seldomly be required to actually do this procedure, since for many packages there are nice tools which make things a charm. (01:11:23 PM) pitti: The bad news is that it may seem utterly complicated for people who never did it before, but I would like you to understand what's actually going on behind the curtains of the tools. (01:11:33 PM) pitti: So please do not desperate if you do not fully understand it at first; there's written documentation and you can always take your time to grok it. (01:11:42 PM) pitti: The general approach is: (01:11:47 PM) pitti: 1. copy the clean source tree to a temporary directory /tmp/old (01:11:52 PM) pitti: 2. apply all patches up to the one you want to edit; if you want to create a new patch, apply all existing ones (this is necessary since in general patches depend on previous patches) (01:12:14 PM) pitti: 3. copy the whole source tree again: cp -a /tmp/old /tmp/new (01:12:18 PM) pitti: 4. go into /tmp/new, do your modifications (01:12:23 PM) pitti: 5. go back into /tmp and generate the patch with (01:12:28 PM) pitti: diff -Nurp old new > mypatchname.patch (01:12:35 PM) pitti: 6. move the newly generated patch to <original source dir>/debian/patches/mypatchname.patch (01:12:44 PM) pitti: in general we want the following diff options: (01:12:50 PM) pitti: -N -> include new files (01:12:50 PM) pitti: -u -> unified patches (context diffs are ugly) (01:12:50 PM) pitti: -r -> recursive (01:12:50 PM) pitti: -p -> bonus, you can see the name of the affected function in the patch (01:13:06 PM) pitti: does anyone have a question about the principle method? (01:13:40 PM) pitti: ok, some hands-on example (01:13:40 PM) pitti: open a shell, ready your fingers :) (01:13:47 PM) pitti: udev example 1, let's create a new patch 92_penguins.patch: (01:13:56 PM) pitti: cd /whereever/you/unpacked/the/source/udev-108 (01:13:56 PM) pitti: -> now we are in our original source tree where we want to add a new patch (01:14:17 PM) pitti: cp -a . /tmp/old (01:14:18 PM) pitti: -> create a copy of the clean sources as reference tree (01:14:27 PM) pitti: pushd /tmp/old (01:14:27 PM) pitti: -> go to /tmp/old; 'pushd' to remember the previous directory, so that we can go back conveniently (01:14:56 PM) pitti: debian/rules patch (01:14:57 PM) pitti: -> apply all already existing patches; of course we could use the 'patch' program to do it manually, but since debian/rules already knows how to do it, let's use it. The actual name for the patch target varies, I have seen the following ones so far: patch, setup, apply-patches, unpack, patch-stamp. You have to look in debian/rules how it is called. (01:15:30 PM) pitti: cp -a . /tmp/new; cd ../new (01:15:30 PM) pitti: -> copies our patched reference tree to our new work directory /tmp/new where we can hack in (01:15:40 PM) pitti: that's the preparatory part (01:15:44 PM) pitti: let's do a braindead modification now (01:15:56 PM) pitti: sed -i 's/Linux/Penguin/g' README (01:15:56 PM) pitti: -> changes the README file; of course you can use your favourite editor, but I wanted to keep my examples copy&pasteable (01:16:12 PM) pitti: and now we create a patch between the reference and our new tree: (01:16:18 PM) pitti: cd .. (01:16:18 PM) pitti: -> go back to /tmp, i. e. where our reference tree (old) and hacked tree (new) is located (01:16:32 PM) pitti: diff -Nurp old new > 95_penguins.patch (01:16:32 PM) pitti: -> generate the patch (Ignore the 'recursive directory loop' warnings) (01:16:56 PM) pitti: popd (01:16:57 PM) pitti: -> now you should be back in your original source tree (when you did the pushd) (01:17:09 PM) pitti: rm -rf /tmp/old /tmp/new (01:17:09 PM) pitti: -> clean up the temporary trees (01:17:17 PM) pitti: mv /tmp/95_penguins.patch debian/patches (01:17:17 PM) pitti: -> move the patch from /tmp to the source tree's patch directory, where it belongs. (01:17:26 PM) pitti: *uff* :) (01:17:34 PM) pitti: Now take a look at your shiny new debian/patches/95_penguins.patch. (01:17:57 PM) pitti: after that, if you do 'debian/rules patch', you'll see that the patch applies cleanly; please do 'debclean' afterwards to unapply the patches and get back a pristine source tree (01:18:17 PM) pitti: so, obviously that's not the end of the wisdom, but if you do these steps a couple of times, you should get a feeling for how to create the most complicated patch conceivable (01:18:29 PM) vattam left the room. (01:18:29 PM) pitti: so this procedure is the life safer if anything else fails (01:18:36 PM) pitti: questions so far? (01:19:27 PM) pitti: are you guys still with me? Am I too fast? (cry and slow me down if so) (01:20:40 PM) ***pitti wonders whether he managed to kill his complete audience now (01:20:45 PM) pitti: but promised, from now on it will get really easy :)
<ranf> COMMENT: debclean complains about missing build dependencies
- ah, right; you can use 'fakeroot debian/rules clean' to work around that, or just apt-get install them
(01:23:44 PM) pitti: Since this case happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose. (01:23:49 PM) pitti: Using this, above steps would reduce to: (01:23:55 PM) pitti: ../dsrc-new-patch 95_penguins.patch (01:23:56 PM) pitti: sed -i 's/Linux/Penguin/g' README (01:23:56 PM) pitti: <press Control-D to leave the subshell> (01:24:13 PM) pitti: that looks slightly better, doesn't it? If you like the script, please put it into your ~/bin, so that it is in your $PATH (01:24:22 PM) pitti: but I had to torture you with the close-to-the-metal method for the sake of understanding. (01:24:43 PM) pitti: dsrc-new-patch is currently too dumb to edit existing patches, or to put patches somewhere else than the top of the patch stack. If you need this, then you need to do the manual approach. (01:25:31 PM) pitti: everyone in sync? (01:27:28 PM) pitti: oh, just answer here, btw (01:27:44 PM) pitti: Loic: just needing more time, or some difficulties?
<Loic> What r the steps done by dsrc-new-patch
- the script does the exact same steps that you did manually before
(01:29:47 PM) pitti: ok, let's go on then (01:29:55 PM) pitti: Since this is so hideously complicated, patch systems were invented to aid you with that. Let's look at the most popular ones now (they are sufficient to allow you to patch about 90% of the archive's source packages; for the rest you have to resort to the manual approach above). (01:30:10 PM) pitti: == pmount: cdbs with simple-patchsys == (01:30:17 PM) pitti: cdbs' simple-patchsys.mk module matches its name, it has no bells and whistles whatsoever. However, it is pretty popular since it is sufficient for most tasks, and long ago I wrote a script 'cdbs-edit-patch' which most people can live with pretty well. This script is contained in the normal cdbs package. (01:30:37 PM) pitti: if you have a package, you can tell that it uses this system by checking debian/rules (01:30:58 PM) pitti: it has 'include /usr/share/cdbs/1/rules/simple-patchsys.mk' (01:31:10 PM) pitti: You just supply the name of a patch to the script, and depending on whether it already exists or not, it will create a new patch or edit an existing one. (01:31:18 PM) pitti: everyone please look in debian/patches, debian/rules to get a feeling how it looks like (01:31:35 PM) pitti: so, let's mess up pmount a bit (01:31:41 PM) pitti: and add a new patch (01:31:49 PM) pitti: cd /whereever/you/unpacked/the/source/pmount-0.9.13 (01:31:49 PM) pitti: cdbs-edit-patch 03-simple-readme.patch (01:31:49 PM) pitti: echo 'This should document pmount' > README (01:31:49 PM) pitti: <press Control-D to leave the subshell> (01:32:01 PM) pitti: easy, isn't it? (01:33:22 PM) pitti: this will take care of applying all patches that need to be applied, can change patches in the middle of the stack, and also create new ones (01:33:34 PM) pitti: Editing an already existing patch works exactly the same way. (01:33:39 PM) pitti: so I won't give a demo (01:33:47 PM) pitti: (BTW, "cdbs-edit-patch" is slightly misleading, since it actually only applies to simple-patchsys.mk. You can also use other cdbs patch system plugins, such as dpatch or quilt.) (01:34:01 PM) pitti: questions? (01:34:53 PM) pitti: == ed: dpatch == (01:34:57 PM) pitti: dpatch is a pretty robust and proven patch system which also ships a script 'dpatch-edit-patch' (01:35:05 PM) pitti: packages which use this build-depend on 'dpatch', and debian/rules includes 'dpatch.mk' (01:35:48 PM) pitti: The two most important things you should be aware of: (01:35:55 PM) pitti: * dpatch does not apply debian/patches/*, but instead applies all patches mentioned in debian/patches/00list, in the mentioned order. That means that you do not have to rely on asciibetical ordering of the patches and can easily disable patches, but you have to make sure to not forget to update 00list if you add a new patch. (01:36:09 PM) pitti: (forgetting to update 00list is a common cause of followup uploads) (01:36:36 PM) ranf: "dpatch.make" that is in debian/rules (01:36:47 PM) pitti: right, sorry (01:36:55 PM) pitti: * dpatch patches are actually scripts that are executed, not just patches fed to 'patch'. That means you can also do fancy things like calling autoconf or using sed in a dpatch if you want. (01:37:04 PM) pitti: using dpatch for non-native patches is rare, and normally you do not need to worry about how a .dpatch file looks like (01:37:11 PM) pitti: but I think it's important to mention it (01:37:19 PM) pitti: so if you ever want to replace *all* instances of Debian with Ubuntu in all files, write a dpatch with a small shell script that uses sed (01:37:26 PM) pitti: instead of doing a 300 KB static patch which won't apply to the next version anyway :) (01:37:34 PM) pitti: The manpage is very good and has examples, too, so I will only give an example here: (01:37:47 PM) pitti: This will edit an already existing patch and take care that all previous patches are applied in order: (01:37:54 PM) pitti: cd /whereever/you/unpacked/the/source/ed-0.2 (01:37:54 PM) pitti: dpatch-edit-patch 05_ed.1-warning-fix (01:37:54 PM) pitti: <edit stuff, press Ctrl+D> (01:38:02 PM) pitti: so that's exactly like cdbs-edit-patch (01:39:14 PM) pitti: ok, now we edited a patch, that was pretty straightforward now (01:39:26 PM) pitti: now let's create a new one; this is a bit different from cdbs-e-p (01:39:43 PM) pitti: dpatch-edit-patch foo.dpatch 06_testsuite-Makefile.dpatch (01:39:43 PM) pitti: <edit stuff, press Ctrl+D> (01:39:43 PM) pitti: echo foo.dpatch >> debian/patches/00list (01:40:05 PM) pitti: This will create a new patch foo.dpatch relative to the already existing 06_testsuite-Makefile.dpatch. If your patch is very confined and does not depend on other patches, you can leave out the second argument. (01:40:31 PM) pitti: please note the last action (adding your new patch at the appropriate position in the patch list) (01:41:01 PM) pitti: is the issue of patch dependencies clear to everyone or do you have questions about it? (01:42:27 PM) pitti: there's a third (and last) common patch system, quilt, but it's a bit more difficult (01:42:42 PM) pitti: do you want to learn about this as well, or rather digest the stuff above and discuss it more in-depth? (01:43:30 PM) jjstwerff: doesn't matter much to me... I will try all of those for myself soon ;) (01:44:24 PM) pitti: then let's discuss the question of patch ordering a bit (01:44:39 PM) pitti: usually, when you maintain packages, there are several types of patches (01:44:58 PM) pitti: sometimes you create a distro-specific patch which will not go upstream (01:45:21 PM) pitti: those should go at the top of the patch stack (01:45:32 PM) pitti: but sometimes you get a bug fix from upstream cvs, etc. (01:45:44 PM) pitti: those should go at the bottom, i. e. be applied to the original code *first* (01:46:11 PM) pitti: so that (a) they have a better chance to actually apply (you might modify the same code in a custom patch) (01:46:31 PM) pitti: and (b) you keep your custom patches up to date, so that you have a better chance to send them to upstream (01:46:40 PM) pitti: upstreams really like patches which apply to their cvs head :) (01:49:07 PM) pitti: if you need a reference of patch systems (including quilt), there is a wiki page https://wiki.ubuntu.com/MOTU/School/PatchingSources which provides most of above information in a more convenient format. (01:49:13 PM) pitti: However, it might be slightly out of date (it's from dapper times). Feel free to update the page and and add missing bits. (01:49:25 PM) pitti: === A glimpse into the future === (01:49:34 PM) pitti: As you saw, Debian source packages do not have any requirements wrt. structure, patch systems, etc., other source package systems like SRPM are much stricter wrt that. This of course means more flexibility, but also much more learning overhead. (01:49:48 PM) pitti: As a member of the security team I can tell tales of the pain of a gazillion different source package layouts... :) (01:50:03 PM) pitti: Therefore some clever people sat together the other day to propose a new design which would both give us a new and unified source package and patch system that uses bzr (with a quilt-like workflow). This would also integrate packages and patches much better into Launchpad and revision control in general. (01:50:24 PM) pitti: Please take a look at https://wiki.ubuntu.com/NoMoreSourcePackages if you are interested in this.
<Loic> Thanks. I can also just edit any file in debian/ as well?
- that's indeed an interesting question. in general I recommend *not* to patch debian/ control files in patches. since developers expect that they can change debian/* stuff at will. patches should really just patching the upstream bits, i. e. everything except debian/*
<Loic> QUESTION: in dpatch-edit-patch foo.dpatch 06_testsuite-Makefile.dpatch why do you have 2 *.dpatch files? Why not just 06_testsuite-Makefile.dpatch ?
- "This will create a new patch foo.dpatch relative to the already existing 06_testsuite-Makefile.dpatch." (i. e. this command will not touch 06_testsuite-Makefile.dpatch *at all*). instead, it will apply all existing patches up to 06_testsuite-Makefile.dpatch and then create a new patch foo.dpatch relative to that with the modifications you do in the subshell
<ranf> So foo lands before or after 06_test...
if you use the 'echo foo.dpatch >> debian/patches/00list' command that I gave, it will land after it. since in dpatch there is no asciibetical patch order, but it is defined in the debian/patches/00list file
<Loic> "and then add the new patch file name at the top in 00list" can I just manually edit 00list after patching?
- yes, as I said, when you create a new dpatch, you *have* to add it to 00list manually. unlike cdbs-edit-patch, where there is no explicit list (i. e. there the list is 'ls debian/patches/*')
<ranf> do you use meld or something like this?
- I don't; maybe you can explain what meld is?
<ranf> a graphical diff.
ah; well, I'm a hopeless hardcore 'diff | vim -' user, I'm afraid
(02:00:23 PM) pitti: if you have some good hints how to use that to make patching more efficient, I invite you to mention it on the wiki page (02:00:34 PM) pitti: but time is up, I'm afraid, so I'll give away the microphone here