Patching1

Revision 4 as of 2008-08-06 16:17:10

Clear message

Dev Week -- Patching Packages -- Martin Pitt -- Mon, Feb 18

see also Friday session.

(11:00:07 AM) pitti: Welcome to the hands-on training session about how to patch Ubuntu source packages!
(11:00:17 AM) pitti: This assumes that you already know what a patch is and how to handle .patch files in general (i. e. for upstream authors).
(11:00:37 AM) pitti: and want to learn how to put them into packages properly, to get them into Ubuntu
(11:00:51 AM) pitti: so, can you please quickly raise your hands if you are participating?
(11:00:56 AM) ***agoliveira here
(11:01:00 AM) anoath: raise
(11:01:04 AM) pitti: just to see how big the audience is
(11:01:05 AM) ***jetsaredim here
(11:01:07 AM) chrome___: raise
(11:01:08 AM) ***andy101 raises hand
(11:01:08 AM) amachu: yep
(11:01:10 AM) ***bfallik raises his
(11:01:11 AM) huats: \o/
(11:01:13 AM) snhome: raises hand
(11:01:13 AM) bananaramax: pong
(11:01:14 AM) KaiserClaudius: here
(11:01:14 AM) pitti: great
(11:01:16 AM) heikki: o/
(11:01:16 AM) sommer: yo
(11:01:17 AM) pitti: wow
(11:01:18 AM) gkulyk: here
(11:01:19 AM) ***Picklesworth raises his hand :)
(11:01:19 AM) Lantash: yeü
(11:01:22 AM) wousser: yes
(11:01:25 AM) bdgraue: ·
(11:01:25 AM) ***pitti is impressed
(11:01:27 AM) InsClusoe: / raises
(11:01:30 AM) fftb: here
(11:01:30 AM) arturodr: hi
(11:01:32 AM) pitti: if anyone has any question, or I'm totally uncomprehensible (sorry for my English, I'm German), please do not hesitate to interrupt and ask *immediately*
(11:01:41 AM) fredre: here
(11:01:42 AM) ***tseliot raises his hand
(11:01:44 AM) ***TrXuk here
(11:01:47 AM) toert: hand
(11:01:52 AM) ***emgent here
(11:02:00 AM) pitti: OTOH, if you have deeper technical questsions, please ask in #-chat
(11:02:06 AM) pitti: Also, don't bother trying to take notes, we'll sort that out at the end. You can fully concentrate on the discussion and examples.
(11:02:09 AM) ***awalton__ quacks.
(11:02:11 AM) pitti: Let's begin with a little bit of history:
(11:02:19 AM) pitti: == Why use separate patches ==
(11:02:24 AM) dholbach: yes, please take your questions to #ubuntu-classroom-chat and prefix them with QUESTION:
(11:02:39 AM) pitti: == Why use separate patches ==
(11:02:48 AM) pitti: In earlier times, people just applied patches inline (i. e. directly in the source code tree). However, this makes it very hard to extract patches later to modify them, send them upstream, etc. Also this means that new upstream versions are a pain, since they generate a lot of rejections when applying the package diff.gz to them.
(11:03:10 AM) pitti: With split-out patches it is much easier to send them upstream, keep track of them, develop them, etc., since you always see which changes belong together.
(11:03:26 AM) pitti: The ideal state is an unmodified tarball from upstream, plus clean and separate patches, plus the packaging bits in debian/. That means that lsdiff -z <sourcepackage>.diff.gz only contains debian/.
(11:03:52 AM) pitti: The first attempts to split-out patches were pretty trivial: storing patches in debian/patches/, and adding some patch/patch -R snippets to debian/rules. This worked for small patches, but provided no tools for editing these patches, updating them for new upstream versions, etc.
(11:04:22 AM) pitti: Thus several standard patch systems were created which are easy to deploy and provide tools for patch juggling and editing.
(11:04:33 AM) pitti: ---
(11:04:47 AM) pitti: What I would like to do now is to introduce the most common patch systems and show some hands-on demo how to add a new patch and how to edit one.
(11:05:02 AM) pitti: For this, I will point at a source package from the current gutsy or hardy archive, quickly explain the patch system, and show how to apply some (braindead) modifications to it. I recommend you to do the same steps in a terminal, so that you get a feeling for the process and can immediately ask questions.
(11:05:16 AM) pitti: is everyone fine with this approach?
(11:05:19 AM) pitti: any questsions so far?
(11:06:04 AM) Square is now known as Squares
(11:06:05 AM) pitti: If you want to try the stuff yourself, please do the following commands (on gutsy) as preparation:
(11:06:12 AM) pitti:   sudo apt-get install dpatch cdbs quilt patchutils devscripts
(11:06:24 AM) pitti: ^ common build tools
(11:06:25 AM) pitti:   apt-get source cron udev pmount ed xterm
(11:06:32 AM) pitti: ^ the example source packages I'll be explaining
(11:06:36 AM) pitti:   wget http://people.ubuntu.com/~pitti/scripts/dsrc-new-patch
(11:06:48 AM) pitti: (I'll talk about this later)
(11:06:51 AM) pitti:   chmod 755 dsrc-new-patch
(11:06:56 AM) pitti: (make my script executable)
(11:06:59 AM) pitti: I deliberately picked the smallest packages I could find
(11:07:03 AM) ***pitti waits a bit for people to do the preparations; any questions so far?
(11:07:35 AM) anoath: nope
(11:07:44 AM) snhome: too fast
(11:08:21 AM) ubuntu_learner: I posted question on ubuntu-classroom-chat on previous Q.
(11:08:28 AM) Kaasethu: ^ common build tools
(11:08:39 AM) pitti: ubuntu_learner: I answered it there, did you see it?
(11:09:03 AM) amachu: Kaasethu: are those that pitti asked to install
(11:09:27 AM) pitti: ah, the ^ means "explanation for the previous line"
(11:09:37 AM) pitti: like an upwards arrow (sorry if that was unclear)
(11:09:38 AM) Kaasethu: thanks -
(11:09:50 AM) Kaasethu: now got it (I think)
(11:10:07 AM) pitti: so, I think most people should have downloaded the lot now?
(11:10:18 AM) pitti: == cron: inline patches ==
(11:10:37 AM) pitti: No patch system at all, nothing much to say about this.  You directly edit the files in the source tree. This is convenient for a simple and quick change, but will bite back for new upstream versions (see above) and is inconvenient for submitting patches upstream, or reviewing for merges.
(11:10:56 AM) pitti: if you do 'lsdiff -z <package>.diff.gz' and you see changes which are not in debian/, then you probably have such a package
(11:11:16 AM) pitti: i. e. that's a good counterexample for what we want to achieve with patch systems
(11:11:35 AM) pitti: all changes are lumped together, not really documented, you don't see which are Debian specific, which are upstream bug fixes, new features, etc.
(11:11:44 AM) pitti: (some KDE packages have autoconf stuff directly in the diff.gz, but that is ok)
(11:11:52 AM) pitti: so, I think I do not need to say anything else about cron, unless someone has a question
(11:12:06 AM) pitti: just look at the diff.gz to see how packaging and code changes are wildly mixed
(11:13:50 AM) pitti: == udev: separate patches, but no standard patch system ==
(11:14:12 AM) pitti: if you look at udev's diff.gz, you see that it only has debian/, and split-out patches in debian/patches
(11:14:15 AM) pitti: one per logical change
(11:14:40 AM) pitti: However, udev is the most complicated case of a "patch system", since you have to do all the hard work for creating/managing patches manually.
(11:14:48 AM) pitti: In order to make you understand what a patch system does, and to give you a fallback method that will *always* work with any patch system, I handle this first.
(11:15:02 AM) pitti: The good news is that you will seldomly be required to actually do this procedure, since for many packages there are nice tools which make things a charm.
(11:15:24 AM) pitti: The bad news is that it may seem utterly complicated for people who never did it before, but I would like you to understand what's actually going on behind the curtains of the tools.
(11:15:44 AM) pitti: So please do not desperate if you do not fully understand it at first; there's written documentation and you can always take your time to grok it.
(11:16:22 AM) ***agoliveira rekons udev is evil
(11:16:24 AM) pitti: The general approach which all patch systems follow, and which you can always do yourself, is:
(11:16:30 AM) pitti: agoliveira: for various reasons :-P
(11:16:36 AM) pitti: 1. copy the clean source tree to a temporary directory /tmp/old
(11:16:41 AM) agoliveira: pitti: Bingo!
(11:16:47 AM) pitti: 2. apply all patches up to the one you want to edit; if you want to create a new patch, apply all existing ones (this is necessary since in general patches depend on previous patches)
(11:17:09 AM) pitti: 3. copy the whole source tree again: cp -a /tmp/old /tmp/new
(11:17:17 AM) pitti: 4. go into /tmp/new, do your modifications
(11:17:32 AM) pitti: 5. go back into /tmp and generate the patch with
(11:17:33 AM) pitti:   diff -Nurp old new > mypatchname.patch
(11:17:44 AM) pitti: 6. move the newly generated patch to <original source dir>/debian/patches/mypatchname.patch
(11:18:08 AM) pitti: don't worry, I'll do an example
(11:18:25 AM) pitti: that's just the general recipe for printing out and hang over your bed/PC/whereever you need it :)
(11:18:36 AM) pitti: in general we want the following diff options:
(11:18:43 AM) pitti: -N -> include new files
(11:18:51 AM) pitti: (which is quite common)
(11:18:54 AM) pitti: -u -> unified patches (context diffs are ugly)
(11:19:02 AM) pitti: -r -> recursive
(11:19:06 AM) pitti: -p -> bonus, you can see the name of the affected function in the patch
(11:19:10 AM) pitti: does anyone have a question about the principle method?
(11:20:19 AM) pitti: ok, some hands-on example
(11:20:23 AM) pitti: open a shell, ready your fingers :)
(11:20:28 AM) pitti: udev example 1, let's create a new patch 92_penguins.patch:
(11:20:34 AM) pitti:   cd /whereever/you/unpacked/the/source/udev-* # -113 on gutsy, -117 on hardy
(11:21:25 AM) pitti: -> now we are in our original source tree where we want to add a new patch
(11:21:31 AM) pitti:   cp -a . /tmp/old
(11:21:35 AM) pitti: -> create a copy of the clean sources as reference tree
(11:21:46 AM) pitti:   pushd /tmp/old
(11:21:50 AM) pitti: -> go to /tmp/old; 'pushd' to remember the previous directory, so that we can go back conveniently
(11:22:05 AM) pitti:   debian/rules patch
(11:22:18 AM) pitti: -> apply all already existing patches; of course we could use the 'patch' program to do it manually, but since debian/rules already knows how to do it, let's use it. The actual name for the patch target varies, I have seen the following ones so far: patch, setup, apply-patches, unpack, patch-stamp. You have to look in debian/rules how it is called.
(11:22:48 AM) pitti: (I have a silly script which just attempts all of those until one succeeds :) )
(11:22:54 AM) pitti:   cp -a . /tmp/new; cd ../new
(11:23:04 AM) pitti: -> copies our patched reference tree to our new work directory /tmp/new where we can hack in
(11:23:13 AM) snhome: got an error with the first patch command
(11:23:17 AM) pitti: oh?
(11:23:27 AM) pitti: (darn, I tested them two hours ago on hardy)
(11:23:37 AM) pitti: snhome: you mean with debian/rules patch?
(11:23:41 AM) snhome: yes
(11:24:00 AM) pitti: (see -chat)
(11:24:11 AM) pitti: let's do a braindead modification now
(11:24:15 AM) pitti:   sed -i 's/Linux/Penguin/g' README
(11:24:22 AM) pitti: -> changes the README file; of course you can use your favourite editor, but I wanted to keep my examples copy&pasteable
(11:24:45 AM) pitti: and now we create a patch between the reference and our new tree:
(11:25:04 AM) pitti: (btw, please cry out in -chat when I'm too fast)
(11:25:13 AM) pitti:   cd ..
(11:25:18 AM) pitti: -> go back to /tmp, i. e. where our reference tree (old) and hacked tree (new) is located
(11:25:29 AM) pitti:   diff -Nurp old new > 95_penguins.patch
(11:25:33 AM) pitti: -> generate the patch (Ignore the 'recursive directory loop' warnings)
(11:26:48 AM) pitti: oops
(11:27:00 AM) pitti: sorry, you need "apt-get install debhelper" if you get errors about dh_testdir
(11:27:11 AM) ***pitti fixes his notes and gives you guys time to catch up
(11:28:30 AM) pitti:   mv /tmp/95_penguins.patch debian/patches
(11:28:34 AM) pitti: -> move the patch from /tmp to the source tree's patch directory, where it belongs.
(11:28:39 AM) pitti: *uff* :)
(11:28:45 AM) pitti: Now take a look at your shiny new debian/patches/95_penguins.patch.
(11:30:25 AM) snhome: error on the mv
(11:30:39 AM) ozanichkovsky: pushd
(11:30:40 AM) jetsaredim: you need to cd back to the udev_xxx dir
(11:30:44 AM) pitti: oops, sorry
(11:30:50 AM) pitti: you have to do "popd" before the mv
(11:31:57 AM) ***pitti wishes there was an "erase previous line" IRC command
(11:32:20 AM) pitti: so, it's "diff", "popd", "mv"
(11:32:42 AM) pitti: please make a noise when you are ready
(11:32:58 AM) jsauer: ready
(11:32:59 AM) fftb: noise
(11:33:06 AM) ozanichkovsky: ready
(11:33:08 AM) Lantas2: ready
(11:33:09 AM) KaiserClaudius: ready
(11:33:18 AM) Picklesworth: peep
(11:33:24 AM) arturodr: ready
(11:33:27 AM) chrome___: ready
(11:33:29 AM) pitti: awesome
(11:33:29 AM) fredre: ready
(11:33:39 AM) pitti: debian/patches/95_penguins.patch looks plausible?
(11:33:46 AM) pitti: after that, if you do 'debian/rules patch', you'll see that the patch applies cleanly; please do 'debclean' afterwards to unapply the patches and get back a pristine source tree
(11:34:53 AM) pitti: if you don't have debclean, install devscripts, or do "fakeroot debian/rules clean" instead
(11:35:14 AM) pitti: it calls the 'clean' target, which means, clean up all built files, unapply patches, etc.
(11:35:28 AM) nareshov: fakeroot worked for me
(11:35:34 AM) pitti: if it complains about missing build dependencies, use the fakeroot command
(11:36:11 AM) pitti: so, obviously that's not the end of the wisdom, but if you do these steps a couple of times, you should get a feeling for how to create the most complicated patch conceivable
(11:36:59 AM) pitti: so this procedure is the life safer if anything else fails
(11:37:33 AM) pitti: Pretty much work, isn't it? Since this happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose.
(11:40:06 AM) Don-1: Sorry, I'm a bit late, so I decided to read from start, and just catch up. However, I'm stuck at debian/rules patch, gives me the error message:
(11:40:06 AM) Don-1: dh_testdir
(11:40:06 AM) Don-1: make: dh_testdir: Command not found
(11:40:06 AM) Don-1: make: *** [patch-stamp] Error 127
(11:40:06 AM) Don-1: I've tried both debian/rules build and all the things you mentioned, but it doesn't seem to work. I've also looked in the file, but it doesn't seem to work...
(11:40:30 AM) nareshov: Need devscripts ?
(11:40:44 AM) dholbach: Don-1: please ask questions in #ubuntu-classroom-chat
(11:40:51 AM) Don-1: Sorry.
(11:40:55 AM) Assid: hi
(11:42:25 AM) dholbach: pitti: you fell of the internet?
(11:42:50 AM) dholbach: minus the missing packages - how did the session go? everything seemed sensible?
(11:43:12 AM) snhome: I'm new at this so will need to study what we did
(11:43:22 AM) dholbach: maybe we can do some general Q&A while pitti recovers from whatever problem he has right now
(11:43:37 AM) ubuntu_learner_: Here or at -chat?
(11:43:40 AM) livewire1 is now known as livewire
(11:43:46 AM) dholbach: http://wiki.ubuntu.com/PackagingGuide/PatchSystems is a guide to patch systems
(11:43:52 AM) dholbach: just ask in #-chat and I'll carry over the questions
(11:44:26 AM) dholbach: <jetsaredim> so, once we patch the package, what then?
(11:44:31 AM) dholbach: good question
(11:44:35 AM) nareshov: :)
(11:44:41 AM) dholbach: in most cases as an Ubuntu developer you work on existing packages
(11:44:42 AM) pitti: argh, sorry; my interweb tube broke
(11:44:46 AM) pitti: what's the last line you saw from me?
(11:44:53 AM) nareshov: 21:53 < pitti> Pretty much work, isn't it? Since this happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose.
(11:44:59 AM) pitti: ugh
(11:45:03 AM) pitti: ok, I'll go on there
(11:45:06 AM) dholbach: let me just reply to jetsaredim
(11:45:10 AM) agoliveira: pitti: it was: "arrrghhhh....."
(11:45:23 AM) fftb: :)
(11:45:48 AM) dholbach: jetsaredim: if you can upload your packages directly, you do that, if you can't you use the sponsorship process to get the patched package uploaded - jetsaredim: I'll talk a bit in my session about debdiffs
(11:45:51 AM) pitti: Using dsrc-new-patch, above steps would reduce to:
(11:46:05 AM) pitti:   ../dsrc-new-patch 95_penguins.patch
(11:46:17 AM) pitti: (.., or wherever you downloaded the script)
(11:46:19 AM) jetsaredim: dholbach: what about changing the package version, etc how does that get determined
(11:46:22 AM) pitti:   sed -i 's/Linux/Penguin/g' README
(11:46:24 AM) pitti:   <press Control-D to leave the subshell>
(11:46:32 AM) Assid left the room ("Who parted?").
(11:46:39 AM) pitti: that looks slightly better, doesn't it? If you like the script, please put it into your ~/bin, so that it is in your $PATH
(11:46:40 AM) dholbach: jetsaredim: I'll cover that in a different session :)
(11:46:45 AM) jetsaredim: ah
(11:46:52 AM) ***jetsaredim returns to lurking
(11:47:01 AM) pitti: but I had to torture you with the close-to-the-metal method for the sake of understanding.
(11:47:48 AM) pitti: You might have noticed that we applied all previous patches before creating our's. Does someone have an idea why this is done?
(11:48:24 AM) jetsaredim: numbering
(11:48:27 AM) nareshov: the patches have numbers?
(11:48:39 AM) pitti: well, it's related to that, yes
(11:48:40 AM) nareshov: sequential, dependant ?
(11:48:43 AM) ***chrome___ thinks that the patches might depend on each other
(11:48:47 AM) bananaramax: <pitti> 2. apply all patches up to the one you want to edit; if you want to create a new patch, apply all existing ones (this is necessary since in general patches depend on previous patches)
(11:48:47 AM) awalton__: asserting patch interdependencies.
(11:48:49 AM) pitti: Patches can depend on each other,
(11:49:00 AM) pitti: i. e. patch 22 can change the same file that patch 7 did, and patch 22 might not even apply to the pristine upstream source.
(11:49:18 AM) pitti: That's why you will commonly see numeric prefixes to the patch files, since they are applied asciibetically in many patch systems (including the application of patches in udev).
(11:49:54 AM) pitti: Since above procedure is so hideously complicated, patch systems were invented to aid you with that.
(11:50:25 AM) pitti: Let's look at the most popular ones now (they are sufficient to allow you to patch about 90% of the archive's source packages; for the rest you have to resort to the manual approach above).
(11:51:04 AM) pitti: == pmount: cdbs with simple-patchsys ==
(11:51:19 AM) pitti: cdbs' simple-patchsys.mk module matches its name, it has no bells and whistles whatsoever. However, it is pretty popular since it is sufficient for most tasks, and long ago I wrote a script 'cdbs-edit-patch' which most people can live with pretty well. This script is contained in the normal cdbs package.
(11:51:55 AM) pitti: Many Ubuntu packages use the cdbs build system; you can look at the "Build-Depends:" line in debian/control (it has cdbs) to check
(11:52:00 AM) Heres is now known as Her0
(11:52:07 AM) pitti: or look in debian/rules whether it has includes for /usr/share/cdbs
(11:52:19 AM) pitti: if you have that, then you can use the following script
(11:52:22 AM) pitti: You just supply the name of a patch to the script, and depending on whether it already exists or not, it will create a new patch or edit an existing one.
(11:53:05 AM) pitti: everyone please look in debian/patches, debian/rules to get a feeling how it looks like
(11:53:11 AM) pitti: so, let's mess up pmount a bit
(11:53:15 AM) pitti: and add a new patch
(11:53:18 AM) pitti:   cd /whereever/you/unpacked/the/source/pmount-0.9.16
(11:53:30 AM) pitti:   cdbs-edit-patch 07-simple-readme.patch
(11:53:46 AM) pitti: (drops you into a subshell for editing again, similar to dsrc-new-patch)
(11:53:50 AM) pitti:   echo 'This should document pmount' > README
(11:53:55 AM) pitti: (some braindead modification)
(11:53:58 AM) pitti:   <press Control-D to leave the subshell>
(11:54:03 AM) pitti: easy, isn't it?
(11:54:14 AM) pitti: this will take care of applying all patches that need to be applied, can change patches in the middle of the stack, and also create new ones
(11:54:32 AM) pitti: Editing an already existing patch works exactly the same way.
(11:54:36 AM) pitti: so I won't give a demo
(11:55:01 AM) pitti: questions?
(11:55:31 AM) pitti: sorry, my falling off the internet costed me 5 minutes, so I'll just quickly run through the other major patch system before I conclude
(11:56:07 AM) pitti: == ed: dpatch ==
(11:56:14 AM) pitti: dpatch is a pretty robust and proven patch system which also ships a script 'dpatch-edit-patch'
(11:56:23 AM) pitti: packages which use this build-depend on 'dpatch', and debian/rules includes 'dpatch.mk'
(11:57:02 AM) pitti: that script dpatch-edit-patch is actually pretty similar to the other two scripts I talked about
(11:57:04 AM) pitti: The two most important things you should be aware of:
(11:57:14 AM) pitti:  * dpatch does not apply debian/patches/*, but instead applies all patches mentioned in debian/patches/00list, in the mentioned order. That means that you do not have to rely on asciibetical ordering of the patches and can easily disable patches, but you have to make sure to not forget to update 00list if you add a new patch.
(11:57:35 AM) pitti: (forgetting to update 00list is a common cause of followup uploads :-P )
(11:58:10 AM) pitti:  * dpatch patches are actually scripts that are executed, not just patches fed to 'patch'. That means you can also do fancy things like calling autoconf or using sed in a dpatch if you want.
(11:58:11 AM) pitti: using dpatch for non-native patches is rare, and normally you do not need to worry about how a .dpatch file looks like
(11:58:20 AM) pitti: but I think it's important to mention it
(11:58:31 AM) pitti: The manpage is very good and has examples, too
(11:59:07 AM) pitti: There is a wiki page https://wiki.ubuntu.com/MOTU/School/PatchingSources which has examples for the most common patch systems
(11:59:09 AM) pitti: and how to use them
(11:59:21 AM) pitti: it's a good cheatsheet when you will actually get to this
(11:59:30 AM) dholbach: pitti: http://wiki.ubuntu.com/PackagingGuide/PatchSystems :)
(11:59:47 AM) pitti: but with above tools you should be able to change the vast majority of packages
(11:59:54 AM) pitti: thanks for your attention!
(12:00:01 PM) pitti: BTW, I'll do the same session on Friday again
(12:00:07 PM) pitti: and hopefully catch up with the other stuff