PatchPackaging2
Ubuntu Open Week - Patching Packages - Thu, Nov 30, 2006
see also Tuesday Session.
06:00 sfllaw Patching Packages with Martin Pitt (pitti) 06:01 pitti so, who are the Happy Patch People around here? :0 === jrib raises hand 06:01 pitti just a warning, today's talk will be 95% the same as the one I held on Tuesday === jorgp raises hand, Im here 06:02 pitti jrib, jorgp: welcome 06:02 ailean me :) 06:02 jrib hmm do you think you can get a chance to cover quilt? 06:02 pitti ah, seems that most of the folks already attended Tuesday 06:02 pitti jrib: yes, if time permits === pitti clears his throat 06:02 pitti if anyone has any question, or I'm totally uncomprehensible (sorry, I'm German), please do not hesitate to interrupt and ask *immediately* 06:02 pitti Also, don't bother trying to take notes, we'll sort that out at the end. You can fully concentrate on the discussion and examples. 06:02 pitti Let's begin with a little bit of history: 06:03 pitti oh, I will usually paste a snippet and then give you some time to catch up and ask, ok? 06:03 pitti == Why use separate patches == 06:03 pitti In earlier times, people just applied patches inline (i. e. directly in the source code tree). However, this makes it very hard to extract patches later to modify them, send them upstream, etc. Also this means that new upstream versions are a pain, since they generate a lot of rejections when applying the package diff.gz to them. 06:03 pitti With split-out patches it is much easier to send them upstream, keep track of them, develop them, etc., since you always see which changes belong together. 06:03 pitti The ideal state is an unmodified tarball from upstream, plus clean and separate patches, plus the packaging bits in debian/. That means that lsdiff <sourcepackage>.diff.gz only contains debian/. 06:04 pitti The first attempts to split-out patches were pretty trivial: storing patches in debian/patches/, and adding some patch/patch -R snippets to debian/rules. This worked for small patches, but provided no tools for editing these patches, updating them for new upstream versions, etc. 06:04 pitti Thus several standard patch systems were created which are easy to deploy and provide tools for patch juggling and editing. 06:04 pitti any general questions at this point? 06:05 pitti What I would like to do now is to introduce the most common patch systems and show some hands-on demo how to add a new patch and how to edit one. For this, I will point at a source package from the current dapper archive, quickly explain the patch system, and show how to apply some (braindead) modifications to it. I recommend you to do the same steps in a terminal, so that you get a feeling for the process and can immediately ask questions. 06:05 pitti everyone you fine with this approach? 06:05 pitti If you want to try the stuff yourself, please do the following commands (on edgy) as preparation: 06:05 pitti sudo apt-get install dpatch cdbs quilt patchutils devscripts 06:05 pitti apt-get source cron udev pmount gnome-volume-manager ed xterm 06:05 pitti wget http://people.ubuntu.com/~pitti/scripts/dsrc-new-patch 06:05 pitti chmod 755 dsrc-new-patch 06:05 pitti I deliberately picked the smallest packages I could find === pitti waits a bit for people to do the preparations 06:08 pitti == cron: inline patches == 06:08 pitti No patch system at all, nothing much to say about this. You directly edit the files in the source tree. This is convenient for a simple and quick change, but will bite back for new upstream versions (see above) and is inconvenient for submitting patches upstream, or reviewing for merges. 06:08 pitti if you do 'lsdiff <package>.diff.gz' and you see changes which are not in debian/, then you probably have such a package 06:08 pitti (some KDE packages have autoconf stuff directly in the diff.gz, but that is ok) 06:08 pitti so, I think I do not need to say anything else about cron, unless someone has a question 06:08 pitti this style of packages can be mainly found in (1) packages which are around for ages (like cron) or packages where Debian/Ubuntu is upstream 06:09 pitti or when the maintainer simply thinks he won't need a patch system 06:09 pitti :) 06:09 pitti ok, then let's dive in 06:09 pitti == udev: separate patches, but no patch system == 06:09 pitti This case is the most complicated one since you have to do all the hard work manually. In order to make you understand what a patch system does, and to give you a fallback method that will *always* work with any patch system, I handle this first. 06:09 pitti The good news is that you will seldomly be required to actually do this procedure, since for many packages there are nice tools which make things a charm. 06:09 pitti The bad news is that it may seem utterly complicated for people who never did it before, but I would like you to understand what's actually going on behind the curtains of the tools. 06:10 pitti since it is your live safer 06:10 pitti life, even 06:10 pitti So please do not desperate if you do not fully understand it at first; there's written documentation and you can always take your time to grok it. 06:10 pitti The general approach is: 06:10 pitti oh, please ask questions right here, as I asked for 06:10 pitti I like to treat them synchronously, since this is a hands-on tutorial
<jrib> we need lsdiff -z <package>.diff.gz right?
- right, that will show what the diff.gz touches. ideally this is just debian/
<ailean> I'm a complete beginner to all of this . . . I'm unsure about what cron is, and what .diff.gz is. Should I consult something else before coming back to read these logs?
- I'm afraid you might be overwhelmed by this stuff. I do assume that you have basic familiarity about what a source pacakge is and how it looks like. this is intended to be a tutorial for people who want to hack on packages, but of course feel free to listen. cron is 'just a random package' for our purposes and a diff.gz is the disto specific part on top of the original upstream tarball, with all the packaging specific bits and code changes, but due to limited time I can't be more specific here
06:13 pitti ok, so the general approach 06:13 pitti 1. copy the clean source tree to a temporary directory /tmp/old 06:13 pitti 2. apply all patches up to the one you want to edit; if you want to create a new patch, apply all existing ones (this is necessary since in general patches depend on previous patches) 06:13 pitti 3. copy the whole source tree again: cp -a /tmp/old /tmp/new 06:13 pitti 4. go into /tmp/new, do your modifications 06:13 pitti 5. go back into /tmp and generate the patch with 06:13 pitti diff -Nurp old new > mypatchname.patch 06:13 pitti 6. move the newly generated patch to <original source dir>/debian/patches/mypatchname.patch 06:14 pitti (just the theory, detailled explanation will follow shortly) 06:14 pitti with the demo it will become clear 06:14 pitti just an explanation of the '-Nurp' diff options: 06:14 pitti in general we want the following diff options: 06:14 pitti -N -> include new files 06:14 pitti -u -> unified patches (context diffs are ugly) 06:14 pitti -r -> recursive 06:14 pitti -p -> bonus, you can see the name of the affected function in the patch 06:14 pitti does anyone have a question about the principle method?
[jrib] when you say we need to apply the patches up to the one we need to edit, does that mean all of the ones that start with a smaller number?
- right. in 95% of the cases, patches are applied in asciibetical order, and since most patches are prefixed with a number, sorted by number
[chickn] wait..before you go too far, i don't get how you know if a package has a 'patching system' or not..
- ah, good question. if there is a debian/patches/ with patches, then most probably there is a patch system. there are very few braindead pacakges which apply their patches inline and additionally put them into debian/patches/ for future reference. this is just *silly*
[chickn] So for cron...there's no reference for 'patch' in rules
- right, and you will see taht lsdiff -z cron*.diff.gz contains source code patches. i. e. it just applies the patches inline in the raw source tree
06:17 pitti and it happened to the best of us that we just sticked a new patch into that and wondered why it didn't become active 06:17 pitti you can check debian/rules and search for 'patch', that's a pretty safe method 06:18 pitti ok, some hands-on example on udev 06:18 pitti open a shell, ready your fingers :) 06:19 pitti udev example 1, let's create a new patch 90_penguins.patch: 06:19 pitti cd /whereever/you/unpacked/the/source/udev-093 06:19 pitti -> now we are in our original source tree where we want to add a new patch 06:20 pitti cp -a . /tmp/old 06:20 pitti -> create a copy of the clean sources as reference tree 06:20 pitti pushd /tmp/old 06:20 pitti -> go to /tmp/old; 'pushd' to remember the previous directory, so that we can go back conveniently 06:20 pitti debian/rules patch 06:20 pitti -> apply all already existing patches; of course we could use the 'patch' program to do it manually, but since debian/rules already knows how to do it, let's use it. The actual name for the patch target varies, I have seen the following ones so far: patch, setup, apply-patches, unpack, patch-stamp. You have to look in debian/rules how it is called. 06:21 pitti cp -a . /tmp/new; cd ../new 06:21 pitti -> copies our patched reference tree to our new work directory /tmp/new where we can hack in 06:21 pitti that's the preparatory part 06:21 pitti set so far? 06:22 pitti let's do a braindead modification now 06:23 pitti sed -i 's/Linux/Penguin/g' README 06:23 pitti -> changes the README file; of course you can use your favourite editor, but I wanted to keep my examples copy&pasteable 06:23 pitti and now we create a patch between the reference and our new tree: 06:23 pitti cd .. 06:23 pitti -> go back to /tmp, i. e. where our reference tree (old) and hacked tree (new) is located 06:23 pitti diff -Nurp old new > 90_penguins.patch 06:23 pitti -> generate the patch (Ignore the 'recursive directory loop' warnings) 06:23 pitti popd 06:23 pitti -> now you should be back in your original source tree (when you did the pushd) 06:24 pitti rm -rf /tmp/old /tmp/new 06:24 pitti -> clean up the temporary trees 06:24 pitti mv /tmp/90_penguins.patch debian/patches 06:24 pitti -> move the patch from /tmp to the source tree's patch directory, where it belongs. 06:24 pitti *uff* :) 06:24 pitti Now take a look at your shiny new debian/patches/90_penguins.patch. 06:24 pitti after that, if you do 'debian/rules patch', you'll see that the patch applies cleanly; please do 'debclean' afterwards to unapply the patches and get back a pristine source tree 06:25 pitti so, obviously that's not the end of the wisdom, but if you do these steps a couple of times, you should get a feeling for how to create the most complicated patch conceivable 06:25 pitti so this procedure is the life safer if anything else fails 06:25 pitti questions so far?
[jrib] I notice that the patches jump from 01 to 40, and sometimes only jump by 5. What is the convention in naming the patches?
- there's no real policy. often patches are named in ascending order, then maybe patch 05 becomes obsolete or applied upstream. sometimes, maintainers organize them in blocks. e. g. in postgresql, upstream patches start with 00, server patches from 10 to 300, contrib patches from 300 to 400, etc. the only rule is that the asciibetical order is the correct order for applying them; they do not even need to start with a number
[chickn] but if you're writing patches on top of ones already applied - shouldn't your name be something that will come last?
- right, usually you should. however, if you pull stuff from upstream, convention is that you order them first, so that the distro-specific patches stay on top. that makes it easier to upgrade to a new upstream version.
06:28 pitti Pretty much work, isn't it? Since this happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose. 06:29 pitti chickn: https://wiki.ubuntu.com/MOTU/School/PatchingSources 06:29 pitti so, please rm debian/patches/90_penguins.patch 06:30 pitti I know, it was precious and hard to do, but now it gets easier :) 06:30 pitti Using this script, above steps would reduce to: 06:30 pitti ../dsrc-new-patch 90_penguins.patch 06:30 pitti sed -i 's/Linux/Penguin/g' README 06:30 pitti <press Control-D to leave the subshell> 06:30 pitti that looks slightly better, doesn't it? If you like the script, please put it into your ~/bin, so that it is in your $PATH 06:30 pitti but I had to torture you with the close-to-the-metal method for the sake of understanding. 06:31 pitti but let's stay at udev a bit and do the second example 06:31 pitti I promise, this will be the last complicated issue in this lesson 06:31 pitti but it might teach you to never ever do a package without a proper patch system :) 06:32 pitti dsrc-new-patch is currently too dumb to edit existing patches, or to put patches somewhere else than the top of the patch stack. If you need this, then you need to do the manual approach. 06:32 pitti Assume we want to edit 50-result-whitespace.patch. This patch does not depend on the previous patches, so it's a bit easier that way, too: 06:32 pitti cd /whereever/you/unpacked/the/source/udev-093 06:32 pitti cp -a . /tmp/old 06:32 pitti pushd /tmp/old 06:32 pitti cp -a . /tmp/new; cd ../new 06:32 pitti -> nothing new so far, just created a reference and a hack tree 06:32 pitti but note that we did not apply any patch (since we assume that 50-result-whitespace.patch does not depend on its predecessors) 06:33 pitti everyone got that? 06:33 pitti patch -Nlp1 < debian/patches/50-result-whitespace.patch 06:33 pitti above will apply the current version of the patch, so that you can edit it 06:34 pitti we have to do that only in /tmp/new, so that our final diff call catches both the old patch and our new modifications 06:34 pitti sed -i '1 s/$/***** HELLO WORLD ****/' udev_selinux.c 06:34 pitti (just some random braindead change) 06:34 pitti now we have the updated tree in /tmp/new 06:34 pitti so let's do the patch: 06:35 pitti cd .. 06:35 pitti diff -Nurp old new > 50-result-whitespace.patch 06:35 pitti popd 06:35 pitti mv /tmp/50-result-whitespace.patch debian/patches 06:35 pitti rm -rf /tmp/old /tmp/new 06:35 pitti exactly the same commands as before 06:35 pitti Now debian/patches/50-result-whitespace.patch contains both the original change and our modification of the first line of the source file. 06:35 pitti but promised, from now on it will get really easy :) 06:35 pitti any questions? is the issue of patch dependencies clear to everyone?
[jrib] if 50-result-whitespace.patch depended on 01-lib-udev.patch for example. Would the procedure then just be to apply 01-lib-udev.patch to old first and then do the same thing as before?
- you got it. if you want to stay at the safe side, just apply all patches before the one you want to edit in /tmp/old. doing that can never hurt
06:36 pitti Since this is so hideously complicated, patch systems were invented to aid you with that. Let's look at the most popular ones now (they are sufficient to allow you to patch about 90% of the archive's source packages; for the rest you have to resort to the manual approach above). 06:37 pitti == pmount: cdbs with simple-patchsys == 06:37 pitti cdbs' simple-patchsys.mk module matches its name, it has no bells and whistles whatsoever. However, it is pretty popular since it is sufficient for most tasks, and long ago I wrote a script 'cdbs-edit-patch' which most people can live with pretty well. This script is contained in the normal cdbs package. 06:37 pitti however, of course it only works for pacakges which actually use cdbs 06:37 pitti i. e. which have a cdbs build-dependency 06:38 pitti and include some stuff (simple-patchsys.mk in particular) from /usr/share/cdbs 06:38 pitti ^ in debian/rules 06:38 pitti please take a quick look at debian/rules in the pmount source 06:38 pitti You just supply the name of a patch to the script, and depending on whether it already exists or not, it will create a new patch or edit an existing one. 06:39 pitti this includes taking care of patch dependencies, editing patches in the middle of the stack, etc. 06:39 pitti so, let's mess up pmount a bit 06:39 pitti and add a new patch 06:39 pitti cd /whereever/you/unpacked/the/source/pmount-0.9.13 06:39 pitti cdbs-edit-patch 03-simple-readme.patch 06:39 pitti echo 'This should document pmount' > README 06:39 pitti <press Control-D to leave the subshell> 06:39 pitti easy, isn't it? 06:40 pitti ok? 06:40 jorgp that was easy 06:40 pitti Editing an already existing patch works exactly the same way. 06:41 pitti so I won't give a demo
[elliot] got to go now. Whats the link for the chat log?
https://wiki.ubuntu.com/MOTU/School/PatchingSources is the wiki page for it
06:41 pitti BTW, "cdbs-edit-patch" is slightly misleading, since it actually only applies to simple-patchsys.mk. You can also use other cdbs patch system plugins, such as dpatch or quilt. 06:41 pitti questions? 06:42 jrib nope, this is a lot easier :) 06:42 pitti I promised :) 06:42 pitti == ed: dpatch == 06:42 pitti dpatch is a pretty robust and proven patch system which also ships a script 'dpatch-edit-patch' 06:42 pitti The two most important things you should be aware of: 06:42 pitti * dpatch does not apply debian/patches/*, but instead applies all patches mentioned in debian/patches/00list, in the mentioned order. That means that you do not have to rely on asciibetical ordering of the patches and can easily disable patches, but you have to make sure to not forget to update 00list if you add a new patch. 06:42 pitti (forgetting to update 00list is a common cause of followup uploads) 06:43 pitti * dpatch patches are actually scripts that are executed, not just patches fed to 'patch'. That means you can also do fancy things like calling autoconf or using sed in a dpatch if you want. 06:43 pitti using dpatch for non-native patches is rare, and normally you do not need to worry about how a .dpatch file looks like 06:43 pitti but I think it's important to mention it 06:43 pitti so if you ever want to replace *all* instances of Debian with Ubuntu in all files, write a dpatch with a small shell script that uses sed 06:43 pitti instead of doing a 300 KB static patch which won't apply to the next version anyway 06:44 pitti The manpage is very good and has examples, too, so I will only give an example here: 06:44 pitti This will edit an already existing patch and take care that all previous patches are applied in order: 06:44 pitti cd /whereever/you/unpacked/the/source/ed-0.2 06:44 pitti dpatch-edit-patch 05_ed.1-warning-fix 06:44 pitti <edit stuff, press Ctrl+D> 06:44 pitti so that's exactly like cdbs-edit-patch 06:44 pitti ok, now we edited a patch, that's pretty easy, right? 06:44 jorgp yes 06:44 pitti now let's create a new one; this is different from cdbs-edit-patch 06:44 pitti (due to the 00list handling) 06:45 pitti dpatch-edit-patch foo.dpatch 06_testsuite-Makefile.dpatch 06:45 pitti <edit stuff, press Ctrl+D> 06:45 pitti echo foo.dpatch >> debian/patches/00list 06:45 pitti This will create a new patch foo.dpatch relative to the already existing 06_testsuite-Makefile.dpatch. If your patch is very confined and does not depend on other patches, you can leave out the second argument. 06:45 pitti please note the last action (adding your new patch at the appropriate position in the patch list) 06:46 pitti so again, dpatch does *not* rely on any asciibetical order by default
[jorgp] would it still be in this scope of discussion to talk about patches that don't apply cleanly
- of course. if you get a source package from the archive, they should apply cleanly and if you got a patch from upstream and stick it into debian/patches/, but it doesn't apply, you have to fix it, of course. cdbs-edit-patch and dpatch-e-p will just try and leave you in the shell with the *.rej files then you simply resolve the *.rej files, and press C-D to update the patch.
however, please keep in mind that a dpatch is a patch with a shell script header, not a simple patch (as mentioned above) so to avoid this entirely, you should never just stick an unknown patch into debian/patches. instead, you use *-edit-patch and *then* use patch -p1 < /path/to/your/new/patch. edit the mess there. and let *-edit-patch create the actual patch in the source tree
06:50 pitti then let's go to the last patch system 06:50 pitti == xterm: quilt == 06:50 pitti quilt is the other non-dumb standard patch system. Like dpatch, it has a list of patches to apply in patches/series (to use debian/patches, packages need to add a sylink). 06:50 pitti It is non-trivial to set up and has a lot of advanced commands which make it very flexible, but not very easy to use. 06:50 pitti nontrivial to set up for Debian source packages, that is 06:50 pitti (it's not hard either, but more work than simple-patchsys, and even dpatch) 06:50 pitti I will only show a small example here 06:50 pitti in the xterm source 06:51 pitti First, you can use the existing machinery to set up symlinks and directories for quilt: 06:51 pitti cd /whereever/you/unpacked/the/source/xterm-216 06:51 pitti debian/rules prepare 06:51 pitti the 'prepare' target is not standardized; you need to look into debian/rules 06:51 pitti however, it usually boils down to 'export QUILT_PATCHES=debian/patches' (which should work fine everywhere) 06:51 pitti since quilt's default patch dir is ./patches 06:51 pitti so some packages set up a symlink, some export QUILT_PATCHES, etc. 06:52 pitti Now let's edit the already existing patch 901_xterm_manpage.diff: 06:52 pitti quilt push 901_xterm_manpage.diff 06:52 pitti this will apply all patches in the stack up to the given one 06:52 pitti apply inline right in the source tree, that is 06:52 pitti i. e. *not* in a temporary directory and not in a subshell 06:52 pitti quilt keeps track of applied patches in the source tree internally 06:53 pitti now let's edit a file that is already touched by the original patch 06:53 pitti sed -i 's/Copyright/Copyleft/' xterm.man 06:53 pitti let's commit the change: 06:53 pitti quilt refresh 901_xterm_manpage.diff 06:53 pitti So unlike the other patch systems, quilt works with patched inline sources, but keeps track of modifications. === pitti waits a bit for people to catch up and finish the example on their keyboards 06:55 pitti Finally, let's add a new patch to the top of the stack: 06:55 pitti quilt push -a 06:55 pitti '-a' means 'all patches', thus it applies all further patches after 901_xterm_manpage.diff up to the top 06:55 pitti quilt new muhaha.diff 06:55 pitti register a new patch name (which we want to put on top of the patch stack) 06:55 pitti quilt add README 06:55 pitti you have to do that for all files you modify, so that quilt can keep track of the original version 06:55 pitti this tells quilt to keep track of the original version of README 06:55 pitti sed -i '1 s/^/MUHAHA/' README 06:55 pitti modify the source 06:56 pitti quilt refresh 06:56 pitti this will finally create debian/patches/muhaha.diff with the changes to README 06:56 pitti as I already said above, quilt has a patch list, too 06:56 pitti in debian/patches/series 06:56 pitti which is much like debian/patches/00list for dpatch 06:56 pitti and if you push -a, then the patch will land on top of the patch stack, and will automatically be put at the end of series 06:57 pitti of course you can create the patch in other levels of the patch stack 06:57 pitti but usually you want the top 06:57 pitti sometimes, when you pull changes from upstream CVS, it's better to put them at the bottom of the stack 06:57 pitti i. e. upstream changes shuold generally come *before* distro-specific changes 06:57 pitti ok? 06:57 jrib ok 06:58 pitti === A glimpse into the future === 06:58 pitti As you saw, Debian source packages do not have any requirements wrt. structure, patch systems, etc., other source package systems like SRPM are much stricter wrt that. This of course means more flexibility, but also much more learning overhead. 06:58 pitti As a member of the security team I can tell tales of the pain of a gazillion different source package layouts... :) 06:58 pitti Therefore some clever people sat together the other day to propose a new design which would both give us a new and unified source package and patch system that uses bzr (with a quilt-like workflow). This would also integrate packages and patches much better into Launchpad and revision control in general. 06:58 pitti Please take a look at https://wiki.ubuntu.com/NoMoreSourcePackages if you are interested in this. 06:59 pitti I don't want to handle this here, since it's really distant future 06:59 pitti oh, and officially again: 06:59 _ion That was new to me. Sounds very cool. 06:59 pitti === Notes === 06:59 pitti The channel topic points to fabbione's logs, so that you can get the full conversation 06:59 pitti There is also a wiki page https://wiki.ubuntu.com/MOTU/School/PatchingSources which provides most of above information in a more convenient format. However, it might be slightly out of date (it's from dapper times). Feel free to update the page and and add missing bits. 06:59 pitti we are already at the end of the our 06:59 pitti thank you for your attention! 06:59 pitti I hope you could learn a bit 07:01 pitti if you have any further questions, can you please mail me? martin.pitt at ubuntu.com
[jrib] sorry, one more question about quilt. After our last command, the patches are still applied so do we have to unapply them before building the source package?
- debclean will take care of that. and if you build without cleaning, then quilt will know that it already applied the patches. as I said, quilt keeps track of everything, it's pretty hard to break
[DenisTheMenace] how can i build the source package?
- debuild -us -uc -b
[jorgp] which patching system do you prefer and why?
I use cdbs+simple-patchsys.mk for my packages. mainly because I like it's simplicity, wrote cdbs-edit-patch which makes it easy to deal with it, and I don't want to use different systems. the concept of factorization and modularization is The Right Thing (tm). if only the docs were better
MeetingLogs/openweekedgy/PatchPackaging2 (last edited 2008-08-06 16:34:53 by localhost)