Ubuntu Open week - Patching Packages - Martin Pitt - Tue, Apr 24, 2007
see also Thursday Session.
(03:02:43 PM) pitti: splendid; sounds like a good size for a hands-on workshop :) (03:02:49 PM) pitti: if anyone has any question, or I'm totally uncomprehensible (sorry for my English, I'm German), please do not hesitate to interrupt and ask *immediately* (03:03:07 PM) pitti: technical questions in #-chat, please (03:03:11 PM) pitti: Also, don't bother trying to take notes, we'll sort that out at the end. You can fully concentrate on the discussion and examples. (03:03:17 PM) pitti: Let's begin with a little bit of history:
<rohan> pitti: just before we start - is it specifically debian/ubuntu packages patching, or source package patching ?
- this seminar is very Debian/Ubuntu specific
(03:04:09 PM) pitti: I assume that you already know what a 'patch' is (03:04:09 PM) pitti: and why you want to do one :) (03:04:09 PM) pitti: == Why use separate patches in Debian/Ubuntu source packages == (03:04:19 PM) pitti: In earlier times, people just applied patches inline (i. e. directly in the source code tree). However, this makes it very hard to extract patches later to modify them, send them upstream, etc. Also this means that new upstream versions are a pain, since they generate a lot of rejections when applying the package diff.gz to them. (03:04:52 PM) pitti: With split-out patches it is much easier to send them upstream, keep track of them, develop them, etc., since you always see which changes belong together. (03:05:00 PM) pitti: The ideal state is an unmodified tarball from upstream, plus clean and separate patches, plus the packaging bits in debian/. That means that lsdiff <sourcepackage>.diff.gz only contains debian/. (03:05:21 PM) pitti: The first attempts to split-out patches were pretty trivial: storing patches in debian/patches/, and adding some patch/patch -R snippets to debian/rules. This worked for small patches, but provided no tools for editing these patches, updating them for new upstream versions, etc. (03:05:40 PM) pitti: Thus several standard patch systems were created which are easy to deploy and provide tools for patch juggling and editing. (03:05:54 PM) pitti: What I would like to do now is to introduce the most common patch systems and show some hands-on demo how to add a new patch and how to edit one. For this, I will point at a source package from the current feisty archive, quickly explain the patch system, and show how to apply some (braindead) modifications to it. I recommend you to do the same steps in a terminal, so that you get a feeling for the process and can immediately ask questions. (03:06:10 PM) pitti: everyone is fine with this approach? (03:07:14 PM) pitti: If you want to try the stuff yourself, please do the following commands (on feisty) as preparation: (03:07:20 PM) pitti: sudo apt-get install dpatch cdbs quilt patchutils devscripts (03:07:20 PM) pitti: apt-get source cron udev pmount gnome-volume-manager ed xterm (03:07:20 PM) pitti: wget http://people.ubuntu.com/~pitti/scripts/dsrc-new-patch (03:07:20 PM) pitti: chmod 755 dsrc-new-patch (03:07:20 PM) pitti: I deliberately picked the smallest packages I could find (03:07:36 PM) ***pitti waits a bit for people to do the preparations; any questions so far?
<habeeb> let's say that we use Gentoo. Can we try it?
- sorry, as I said this stuff only applies to packaging Debian-style source packages
(03:10:10 PM) pitti: == cron: inline patches == (03:10:23 PM) pitti: No patch system at all, nothing much to say about this. You directly edit the files in the source tree. This is convenient for a simple and quick change, but will bite back for new upstream versions (see above) and is inconvenient for submitting patches upstream, or reviewing for merges. (03:10:38 PM) pitti: if you do 'lsdiff <package>.diff.gz' and you see changes which are not in debian/, then you probably have such a package (03:10:58 PM) pitti: (some KDE packages have autoconf stuff directly in the diff.gz, but that is ok) (03:11:04 PM) pitti: so, I think I do not need to say anything else about cron, unless someone has a question?
<shiyee> lsdiff cron_3.0pl1-100ubuntu1.diff.gz doesn't give me any output...
- lsdiff -z
<rohan> pitti: i did not understand the KDE thing
- KDE has a rule called 'buildprep' which automatically updates configure, Makefile.in etc. from configure.ac/Makefile.am etc. those are entirely autogenerated changes, so developers do not need to maintain these manually
(03:13:26 PM) pitti: == udev: separate patches, but no standard patch system == (03:13:32 PM) pitti: This case is the most complicated one since you have to do all the hard work manually. In order to make you understand what a patch system does, and to give you a fallback method that will *always* work with any patch system, I handle this first. (03:13:50 PM) pitti: The good news is that you will seldomly be required to actually do this procedure, since for many packages there are nice tools which make things a charm. (03:14:01 PM) pitti: The bad news is that it may seem utterly complicated for people who never did it before, but I would like you to understand what's actually going on behind the curtains of the tools. (03:14:20 PM) pitti: So please do not desperate if you do not fully understand it at first; there's written documentation and you can always take your time to grok it. (03:14:30 PM) pitti: The general approach is: (03:14:40 PM) pitti: 1. copy the clean source tree to a temporary directory /tmp/old (03:14:40 PM) pitti: 2. apply all patches up to the one you want to edit; if you want to create a new patch, apply all existing ones (this is necessary since in general patches depend on previous patches) (03:14:40 PM) pitti: 3. copy the whole source tree again: cp -a /tmp/old /tmp/new (03:14:40 PM) pitti: 4. go into /tmp/new, do your modifications (03:14:40 PM) pitti: 5. go back into /tmp and generate the patch with (03:14:42 PM) pitti: diff -Nurp old new > mypatchname.patch (03:14:44 PM) pitti: 6. move the newly generated patch to <original source dir>/debian/patches/mypatchname.patch (03:15:23 PM) pitti: in general we want the following diff options: (03:15:23 PM) pitti: -N -> include new files (03:15:23 PM) pitti: -u -> unified patches (context diffs are ugly) (03:15:23 PM) pitti: -r -> recursive (03:15:23 PM) pitti: -p -> bonus, you can see the name of the affected function in the patch (03:15:36 PM) pitti: does anyone have a question about the principle method?
<rohan> how would the patch system know in what order to apply "mypatchname.patch" ?
- most patch systems just use asciibetical ordering of files in debian/patches. there are some notable exceptions, I will mention them later. but you can usually rely on the rule that whenever you need this by-hand approach it is asciibetical
<crevette> so you need to prefix it with 10- 20-
- that's the common practice, yes . much like in good old BASIC days :-P
(03:17:26 PM) pitti: ok, some hands-on example (03:17:26 PM) pitti: open a shell, ready your fingers :) (03:17:31 PM) pitti: udev example 1, let's create a new patch 92_penguins.patch: (03:17:38 PM) pitti: cd /whereever/you/unpacked/the/source/udev-108 (03:17:41 PM) pitti: -> now we are in our original source tree where we want to add a new patch (03:17:59 PM) pitti: cp -a . /tmp/old (03:17:59 PM) pitti: -> create a copy of the clean sources as reference tree (03:18:12 PM) pitti: pushd /tmp/old (03:18:15 PM) pitti: -> go to /tmp/old; 'pushd' to remember the previous directory, so that we can go back conveniently (03:18:32 PM) pitti: debian/rules patch (03:18:34 PM) pitti: -> apply all already existing patches; of course we could use the 'patch' program to do it manually, but since debian/rules already knows how to do it, let's use it. The actual name for the patch target varies, I have seen the following ones so far: patch, setup, apply-patches, unpack, patch-stamp. You have to look in debian/rules how it is called. (03:19:05 PM) pitti: cp -a . /tmp/new; cd ../new (03:19:05 PM) pitti: -> copies our patched reference tree to our new work directory /tmp/new where we can hack in (03:19:14 PM) pitti: that's the preparatory part (03:19:14 PM) pitti: let's do a braindead modification now (03:19:26 PM) pitti: sed -i 's/Linux/Penguin/g' README (03:19:27 PM) pitti: -> changes the README file; of course you can use your favourite editor, but I wanted to keep my examples copy&pasteable (03:19:44 PM) pitti: and now we create a patch between the reference and our new tree: (03:19:53 PM) pitti: cd .. (03:19:53 PM) pitti: -> go back to /tmp, i. e. where our reference tree (old) and hacked tree (new) is located (03:20:02 PM) pitti: diff -Nurp old new > 95_penguins.patch (03:20:02 PM) pitti: -> generate the patch (Ignore the 'recursive directory loop' warnings) (03:20:20 PM) pitti: btw, NB that we need to be in /tmp for that (03:20:43 PM) pitti: so that the directories in the patch start with 'old/' and 'new/' (patchlevel 1, which is most common) (03:20:58 PM) pitti: popd (03:20:58 PM) pitti: -> now you should be back in your original source tree (when you did the pushd) (03:21:17 PM) pitti: rm -rf /tmp/old /tmp/new (03:21:17 PM) pitti: -> clean up the temporary trees (03:21:20 PM) pitti: mv /tmp/95_penguins.patch debian/patches (03:21:20 PM) pitti: -> move the patch from /tmp to the source tree's patch directory, where it belongs. (03:21:25 PM) pitti: *uff* :) (03:21:30 PM) pitti: Now take a look at your shiny new debian/patches/95_penguins.patch. (03:21:54 PM) pitti: after that, if you do 'debian/rules patch', you'll see that the patch applies cleanly; please do 'debclean' afterwards to unapply the patches and get back a pristine source tree (03:22:16 PM) pitti: so, obviously that's not the end of the wisdom, but if you do these steps a couple of times, you should get a feeling for how to create the most complicated patch conceivable (03:22:28 PM) pitti: so this procedure is the life safer if anything else fails (03:22:34 PM) pitti: questions? (03:23:07 PM) ***pitti wonders whether he managed to kill his complete audience now (03:23:12 PM) pitti: but promised, from now on it will get really easy :) (03:23:28 PM) pitti: Pretty much work, isn't it? Since this happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose. Using this, above steps would reduce to:
<sebasgro> how do i do debclean?
- it's contained in the 'devscripts' package.
(03:27:05 PM) ***pitti will go back a step, just in case (03:27:14 PM) pitti: Since this happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose. Using this, above steps would reduce to: (03:27:14 PM) pitti: ../dsrc-new-patch 95_penguins.patch (03:27:14 PM) pitti: sed -i 's/Linux/Penguin/g' README (03:27:14 PM) pitti: <press Control-D to leave the subshell> (03:27:26 PM) pitti: that looks slightly better, doesn't it? If you like the script, please put it into your ~/bin, so that it is in your $PATH (03:27:35 PM) pitti: but I had to torture you with the close-to-the-metal method for the sake of understanding. (03:28:23 PM) pitti: I have a second example prepared which changes an existing patch, but I'll spare you that thing; if you are interested, we can talk about it afterwards (03:28:39 PM) pitti: Since this is so hideously complicated, patch systems were invented to aid you with that. Let's look at the most popular ones now (they are sufficient to allow you to patch about 90% of the archive's source packages; for the rest you have to resort to the manual approach above). (03:29:00 PM) pitti: == pmount: cdbs with simple-patchsys == (03:29:13 PM) pitti: cdbs' simple-patchsys.mk module matches its name, it has no bells and whistles whatsoever. However, it is pretty popular since it is sufficient for most tasks, and long ago I wrote a script 'cdbs-edit-patch' which most people can live with pretty well. This script is contained in the normal cdbs package. (03:29:31 PM) pitti: You just supply the name of a patch to the script, and depending on whether it already exists or not, it will create a new patch or edit an existing one. (03:29:43 PM) pitti: everyone please look in debian/patches, debian/rules to get a feeling how it looks like (03:30:22 PM) pitti: so, let's mess up pmount a bit and add a new patch (03:30:29 PM) pitti: cd /whereever/you/unpacked/the/source/pmount-0.9.13 (03:30:29 PM) pitti: cdbs-edit-patch 03-simple-readme.patch (03:30:29 PM) pitti: echo 'This should document pmount' > README (03:30:29 PM) pitti: <press Control-D to leave the subshell> (03:30:40 PM) pitti: easy, isn't it? (03:31:11 PM) pitti: this will take care of applying all patches that need to be applied, can change patches in the middle of the stack, and also create new ones (03:31:25 PM) pitti: Editing an already existing patch works exactly the same way. (03:31:31 PM) pitti: so I won't give a demo (03:31:37 PM) pitti: BTW, "cdbs-edit-patch" is slightly misleading, since it actually only applies to simple-patchsys.mk. You can also use other cdbs patch system plugins, such as dpatch or quilt. (03:31:49 PM) pitti: questions? (03:32:22 PM) rohan: what is "cdbs" ? what does it stand for ? (03:32:36 PM) pitti: it's spelled out 'common Debian build system' (03:32:48 PM) pitti: it provides template code for debian/rules (03:33:19 PM) pitti: i. e. common things like 'use autoconf for configuring, make install for shuffling the files around, use intltool to build .pot files, and integrate patch systems (03:33:40 PM) EADG_ is now known as EADG (03:33:43 PM) pitti: with it, you can build source packages in a very high-level way (03:34:30 PM) pitti: for the purpose of this seminar, it's the build system that is used for Gnome and KDE packages (and others, of course) (03:34:42 PM) tseliot: Which patch system should I use to make a patch (editing only 2 files) for the linux-restricted-modules (e.g. so as to solve a problem on launchpad)? (03:35:32 PM) pitti: tseliot: this package does not have any patch system (03:35:44 PM) pitti: tseliot: so this falls into the 'cron' class of source packages (03:36:16 PM) pitti: ok, let's go on then (03:36:21 PM) pitti: == ed: dpatch == (03:36:28 PM) pitti: dpatch is a pretty robust and proven patch system which also ships a script 'dpatch-edit-patch' (03:36:57 PM) pitti: packages which use this build-depend on 'dpatch', and debian/rules includes 'dpatch.mk' (03:37:08 PM) pitti: The two most important things you should be aware of: (03:37:26 PM) pitti: * dpatch does not apply debian/patches/*, but instead applies all patches mentioned in debian/patches/00list, in the mentioned order. That means that you do not have to rely on asciibetical ordering of the patches and can easily disable patches, but you have to make sure to not forget to update 00list if you add a new patch. (03:37:46 PM) pitti: (forgetting to update 00list is a common cause of followup uploads :-) ) (03:37:59 PM) pitti: * dpatch patches are actually scripts that are executed, not just patches fed to 'patch'. That means you can also do fancy things like calling autoconf or using sed in a dpatch if you want. (03:38:15 PM) pitti: using dpatch for non-native patches is rare, and normally you do not need to worry about how a .dpatch file looks like (03:38:23 PM) pitti: but I think it's important to mention it (03:38:32 PM) pitti: so if you ever want to replace *all* instances of Debian with Ubuntu in all files, write a dpatch with a small shell script that uses sed :) (03:39:01 PM) pitti: instead of doing a 300 KB static patch which won't apply to the next version anyway (03:39:09 PM) pitti: The manpage is very good and has examples, too, so I will only give an example here: (03:39:18 PM) pitti: This will edit an already existing patch and take care that all previous patches are applied in order: (03:39:28 PM) pitti: cd /whereever/you/unpacked/the/source/ed-0.2 (03:39:29 PM) pitti: dpatch-edit-patch 05_ed.1-warning-fix (03:39:29 PM) pitti: <edit stuff, press Ctrl+D> (03:39:50 PM) pitti: so that's exactly like cdbs-edit-patch (03:40:03 PM) pitti: ok, now we edited a patch, that's pretty easy, right? (03:40:50 PM) pitti: can you follow at this speed or shall I slow down? (03:41:08 PM) pitti: great (03:41:12 PM) pitti: now let's create a new one; this is different from cdbs-e-p (03:41:22 PM) pitti: due to the 00list thingy I mentioned above (03:41:28 PM) pitti: dpatch-edit-patch foo.dpatch 06_testsuite-Makefile.dpatch (03:41:28 PM) pitti: <edit stuff, press Ctrl+D> (03:41:28 PM) pitti: echo foo.dpatch >> debian/patches/00list (03:41:44 PM) pitti: ^^ NB the last command to update the patch list (you can also use a normal editor, of course) (03:41:56 PM) pitti: This will create a new patch foo.dpatch relative to the already existing 06_testsuite-Makefile.dpatch. If your patch is very confined and does not depend on other patches, you can leave out the second argument. (03:42:02 PM) pitti: alright? (03:42:17 PM) pitti: is the problem of patch dependencies clear to everyone?
<sebasgro> yes but is there a way to see these dependencies?
- it's hard to 'visualize them'. you can look at the patches and check if they patch the same files at roughly the same position
<richb> If you have a mess of several dependent patches is it sometimes best to merge into one?
- depends. of course it does not make sense to fix a feature with a bugfix a, and the next patch reverts that and uses fix b instead. but sometimes patch 1 provides some new infrastructure, and patch 2 uses that new infrastructure to provide a new feature. and you might want to keep the patches separate because patch 1 might go upstream, but patch 2 doesn't have a chance.
<allee> dpatch updates patch following 06_testsuite... if they are affected?
- good point; no, it won't. so if you edit a patch in the middle of a stack, you can theoretically break the patches further up. ('up' in stack order, i. e. the patches further down in 00list)
<allee> is there a tool to adpated such pathces?
- there can't be; if two patches conflict, then only humans can resolve this. you can do 'dpatch-edit-patch 08_conflicting_patch'. this will put you into a subshell again with some .rej files. you resolve them, Ctrl+D, and it will be good again
<allee> e.g. offset fixes and adding conflicts markers to fix by hand
- offset fixes aren't strictly necessary, they don't break patches in general. no SVN-like conflict markers, just the patch-typical .rej files
(03:47:57 PM) pitti: ok, let's move on and handle further questions afterwards (03:48:09 PM) pitti: let's go to the last patch system (03:48:12 PM) pitti: == xterm: quilt == (03:48:16 PM) pitti: quilt is the other non-dumb standard patch system. Like dpatch, it has a list of patches to apply in patches/series (to use debian/patches, packages need to add a sylink). (03:48:28 PM) pitti: It is non-trivial to set up and has a lot of advanced commands which make it very flexible, but not very easy to use. (03:48:38 PM) pitti: nontrivial to set up for Debian source packages, that is (03:48:44 PM) pitti: (it's not hard either, but more work than simple-patchsys, and even dpatch) (03:48:59 PM) pitti: I will only show a small example here (03:49:57 PM) pitti: in the xterm source (03:49:57 PM) pitti: First, you can use the existing machinery to set up symlinks and directories for quilt: (03:50:15 PM) pitti: cd /whereever/you/unpacked/the/source/xterm-223 (03:50:15 PM) pitti: debian/rules prepare (03:50:15 PM) pitti: the 'prepare' target is not standardized; you need to look into debian/rules; however, it usually boils down to 'export QUILT_PATCHES=debian/patches' (which should work fine everywhere) (03:50:15 PM) pitti: (since quilt looks in ./patches by default) (03:50:18 PM) pitti: but I highly recommend to *not* use ./patches for source packages (03:50:20 PM) pitti: all distro changes should be below debian/ (03:50:26 PM) pitti: Now let's edit the already existing patch 901_xterm_manpage.diff: (03:50:35 PM) pitti: quilt push 901_xterm_manpage.diff (03:50:36 PM) pitti: this will apply all patches in the stack up to the given one (03:50:44 PM) pitti: apply inline right in the source tree, that is (03:50:51 PM) pitti: now let's edit a file that is already touched by the original patch (03:50:57 PM) pitti: sed -i 's/Copyright/Copyleft/' xterm.man (03:51:10 PM) pitti: (yay for my creative braindead changes :) )
<rohan> but quilt depends on just then initial number in the patchname for the order ? or does it use some other logic ?
- as I wrote above, quilt uses debian/patches/series, similar to dpatch's 00list. so again you don't need to worry about asciibetical ordering. however, it's common practice to still give them number prefixes. simply because it's better to understand and sort when looking at the patches/ dir. (and incidentally you can actually teach quilt to not use 'series', but debian/patches/*)
(03:53:01 PM) pitti: let's commit the change: (03:53:08 PM) pitti: quilt refresh 901_xterm_manpage.diff (03:53:08 PM) pitti: quilt pop -a (03:53:24 PM) pitti: the latter will 'unwind' all the applied patches, so that you are back to a pristine source tree (03:53:37 PM) pitti: So unlike the other patch systems, quilt works with patched inline sources, but keeps track of modifications. (03:53:39 PM) ***pitti waits a bit for people to catch up and finish the example on their keyboards (03:54:00 PM) pitti: ok everyone? (03:54:33 PM) pitti: Finally, let's add a new patch to the top of the stack: (03:54:41 PM) pitti: quilt push -a (03:54:41 PM) pitti: '-a' means 'all patches', thus it applies all further patches after 901_xterm_manpage.diff up to the top (03:54:47 PM) ***\sh has a special question, but I think it's better asked at the end of this session :) (03:54:58 PM) pitti: \sh: queue it in -chat, plz (03:55:10 PM) pitti: quilt new muhaha.diff (03:55:10 PM) pitti: register a new patch name (which we want to put on top of the patch stack) (03:55:25 PM) pitti: quilt add README (03:55:25 PM) pitti: you have to do that for all files you modify, so that quilt can keep track of the original version (03:55:25 PM) pitti: this tells quilt to keep track of the original version of README (03:55:39 PM) pitti: sed -i '1 s/^/MUHAHA/' README (03:55:39 PM) pitti: modify the source (03:55:52 PM) pitti: quilt refresh (03:55:53 PM) pitti: quilt pop -a (03:55:53 PM) pitti: this will finally create debian/patches/muhaha.diff with the changes to README (03:56:08 PM) pitti: as I already said above, quilt has a patch list, too (03:56:08 PM) pitti: in debian/patches/series (03:56:08 PM) pitti: which is much like debian/patches/00list for dpatch (03:56:21 PM) pitti: and if you push -a, then the patch will land on top of the patch stack, and will automatically be put at the end of series (03:56:37 PM) pitti: of course you can create the patch in other levels of the patch stack (03:56:37 PM) pitti: but usually you want the top (03:56:51 PM) pitti: sometimes, when you pull changes from upstream CVS, it's better to put them at the bottom of the stack (03:56:51 PM) pitti: i. e. upstream changes shuold generally come *before* distro-specific changes (03:57:14 PM) pitti: === A glimpse into the future === (03:57:27 PM) pitti: As you saw, Debian source packages do not have any requirements wrt. structure, patch systems, etc., other source package systems like SRPM are much stricter wrt that. This of course means more flexibility, but also much more learning overhead. (03:57:40 PM) pitti: As a member of the security team I can tell tales of the pain of a gazillion different source package layouts... :) (03:57:58 PM) pitti: Therefore some clever people sat together the other day to propose a new design which would both give us a new and unified source package and patch system that uses bzr (with a quilt-like workflow). This would also integrate packages and patches much better into Launchpad and revision control in general. (03:58:11 PM) pitti: Please take a look at https://wiki.ubuntu.com/NoMoreSourcePackages if you are interested in this. (03:58:19 PM) pitti: not important here, but interesting to mention :) (03:58:45 PM) pitti: erk, we are close to the end, I'll be around for a while in -chat for any further questions (03:58:59 PM) pitti: NOTES: (03:59:10 PM) pitti: THere is a iki page https://wiki.ubuntu.com/MOTU/School/PatchingSources which provides most of above information in a more convenient format. However, it might be slightly out of date (it's from dapper times). Feel free to update the page and and add missing bits. (03:59:30 PM) pitti: THanks for your attention! I hope it was a bit useful (03:59:32 PM) pitti: and happy patching!