Ubuntu Open Week - Patching Packages - Tue, Nov 28, 2006
see also Thursday Session.
08:00 pitti WELCOME TO MY CLASS 08:01 pitti I have a rather technical presentation now, for everyone who wants to modify anything in Ubuntu 08:01 pitti just as a quick overview, who is here to learn about patching? 08:02 pitti great! 08:02 pitti ok, time's limited, let's start 08:02 pitti if anyone has any question, or I'm totally uncomprehensible (sorry, I'm German), please do not hesitate to interrupt and ask *immediately* 08:02 pitti Also, don't bother trying to take notes, we'll sort that out at the end. You can fully concentrate on the discussion and examples. 08:02 pitti Let's begin with a little bit of history: 08:02 pitti == Why use separate patches == 08:02 pitti In earlier times, people just applied patches inline (i. e. directly in the source code tree). However, this makes it very hard to extract patches later to modify them, send them upstream, etc. Also this means that new upstream versions are a pain, since they generate a lot of rejections when applying the package diff.gz to them. 08:02 pitti With split-out patches it is much easier to send them upstream, keep track of them, develop them, etc., since you always see which changes belong together. 08:02 pitti The ideal state is an unmodified tarball from upstream, plus clean and separate patches, plus the packaging bits in debian/. That means that lsdiff <sourcepackage>.diff.gz only contains debian/. 08:02 pitti The first attempts to split-out patches were pretty trivial: storing patches in debian/patches/, and adding some patch/patch -R snippets to debian/rules. This worked for small patches, but provided no tools for editing these patches, updating them for new upstream versions, etc. 08:02 pitti Thus several standard patch systems were created which are easy to deploy and provide tools for patch juggling and editing. 08:04 pitti What I would like to do now is to introduce the most common patch systems and show some hands-on demo how to add a new patch and how to edit one. For this, I will point at a source package from the current dapper archive, quickly explain the patch system, and show how to apply some (braindead) modifications to it. I recommend you to do the same steps in a terminal, so that you get a feeling for the process and can immediately ask questions. 08:04 pitti everyone you fine with this approach? 08:04 pitti erm, of course I mean 'from the edgy archive', not dapper 08:04 pitti yeah, I don't want it to be entirely theoretical :) 08:05 pitti If you want to try the stuff yourself, please do the following commands (on edgy) as preparation: 08:05 pitti sudo apt-get install dpatch cdbs quilt patchutils 08:05 pitti apt-get source cron udev pmount gnome-volume-manager ed xterm 08:05 pitti wget http://people.ubuntu.com/~pitti/scripts/dsrc-new-patch 08:05 pitti chmod 755 dsrc-new-patch 08:05 pitti I deliberately picked the smallest packages I could find 08:08 pitti ok, everyone set with the intro and getting source packages? 08:09 pitti nevermind, for the first bit we don't yet need the packages 08:09 pitti == cron: inline patches == 08:09 pitti No patch system at all, nothing much to say about this. You directly edit the files in the source tree. This is convenient for a simple and quick change, but will bite back for new upstream versions (see above) and is inconvenient for submitting patches upstream, or reviewing for merges. 08:09 pitti if you do 'lsdiff <package>.diff.gz' and you see changes which are not in debian/, then you probably have such a package 08:09 pitti (some KDE packages have autoconf stuff directly in the diff.gz, but that is ok) 08:09 pitti so, I think I do not need to say anything else about cron, unless someone has a question 08:10 pitti ok, no question here 08:10 pitti so, people, get a grip, now it gets heavy 08:11 pitti == udev: separate patches, but no patch system == 08:11 pitti This case is the most complicated one since you have to do all the hard work manually. In order to make you understand what a patch system does, and to give you a fallback method that will *always* work with any patch system, I handle this first. 08:11 pitti The good news is that you will seldomly be required to actually do this procedure, I'll come to this again. 08:11 pitti The general approach is: 08:11 pitti 1. copy the clean source tree to a temporary directory /tmp/old 08:11 pitti 2. apply all patches up to the one you want to edit; if you want to create a new patch, apply all existing ones (this is necessary since in general patches depend on previous patches) 08:11 pitti if you want, you can use debian/rules for this: remove the patches that come *after* the one you want to edit, and call 'debian/rules patch'. The actual name for the patch target varies, I have seen the following ones so far: patch setup apply-patches unpack patch-stamp. You have to look in debian/rules how it is called. 08:11 pitti 3. copy the whole source tree again: cp -a /tmp/old /tmp/new 08:11 pitti 4. go into /tmp/new, do your modifications 08:11 pitti 5. go back into /tmp generate the patch with 08:11 pitti diff -Nurp old new > mypatchname.patch 08:11 pitti and move the patch to .../debian/patches/mypatchname.patch 08:11 pitti ^ that's the theory, example will follow shortly (hands-on training)
[jonh_wendell] pitti, i do a lsdiff <package>.diff and got nothing
- that won't work in that form - lsdiff -z package.diff.gz. -z is for 'accept gzipped files', source packages have a diff.gz, not a .diff
08:12 pitti these steps are the general method how any patch is handled 08:14 pitti in general we want the following diff options: 08:14 pitti -N -> include new files 08:14 pitti -u -> unified patches (context diffs are ugly) 08:14 pitti -r -> recursive 08:14 pitti -p -> bonus, you can see the name of the affected function in the patch 08:14 pitti does anyone have a question about the principle method? 08:14 pitti (I know it's heavy, sorry, that's live; promised, it'll get easier)
[jonh_wendell] pitti, about cron, are you showing a wrong way to package?
- well, it's not exactly 'wrong' as long as the package is not heavily modified, or Debian/Ubuntu is essentially the upstream, it's fine, but having all patches in a single big lump makes it next to impossible to send stuff upstream, or tell which bit belongs to what. also, the tendency is that packages which have 'always' been there, have more 'archaic' build systems and no patch system where as most recently packaged stuff uses a standard packaging system with a standard patch system, now that they are invented
[mwicki] does anybody know how to install a debianpaket whitout root-permission, is it possible to install a debian-paket in a directory??
[Jon] you can unpack it using dpkg-deb -x <package> <directory> as any user who can write to <directory>. mwicki: be warned, though, that doesn't run the pre or post install scripts do you need to do that for patching? it most probably won't work. if it's a simple program then you could then run the binaries in it
08:17 pitti ok, let's go back to patching udev - an EXAMPLE! \o/ 08:17 pitti open a shell, ready your fingers :) 08:17 pitti udev example 1, let's create a new patch 90_penguins.patch: 08:17 pitti cd /whereever/you/unpacked/the/source/udev-093 08:17 pitti cp -a . /tmp/old 08:17 pitti pushd /tmp/old 08:17 pitti debian/rules patch 08:17 pitti cp -a . /tmp/new; cd ../new 08:17 pitti that's the preparatory part, now we have a directory we can hack in 08:18 pitti let's do a braindead modification 08:18 pitti sed -i 's/Linux/Penguin/g' README 08:18 pitti ^ of course you can also use a normal editor 08:18 pitti I just use sed so that you can copy&paste the commands without any interactive stuff 08:19 pitti and now we create a patch between the reference and our new tree 08:19 pitti cd .. 08:19 pitti diff -Nurp old new > 90_penguins.patch 08:19 pitti popd 08:19 pitti rm -rf /tmp/old /tmp/new 08:19 pitti mv /tmp/90_penguins.patch debian/patches 08:19 pitti (Ignore the 'recursive directory loop' warnings)
[jonh_wendell] what does pushd do?
- 'pushd foo' is like 'cd foo', but remembers the last dir and pushes it onto a stack. if you then do 'popd', it will return to the directory you have been in when you did 'pushd'
[StaCk] isn't it like cd - ? no, that will go only to the previous dir. you can stack multiple pushd's on top of each other, and allow 'cd' in between
08:21 pitti Now take a look at your shiny new debian/patches/90_penguins.patch. 08:21 pitti *uff* :) 08:21 pitti after that, if you do 'debian/rules patch', you'll see that the patch applies cleanly 08:21 pitti 08:22 pitti so, obviously that's not the end of the wisdom, but if you do these steps a couple of times, you should get a feeling for how to create the most complicated patch conceivable 08:22 pitti so this procedure is the life safer if anything else fails 08:22 pitti questions so far?
[pschulz01] pitti: cd new?
- 'cp -a . /tmp/new; cd ../new' We applied all previous patches in /tmp/old, then copied /tmp/old to /tmp/new, and hacked our stuff in /tmp/new
[dieterd] result is then a new cd with included patches?
- the result after all of these steps is a single new file debian/patches/90_penguins.patch. the commands above removed the temporary /tmp/old and /tmp/new trees
[pschulz01] Need to copy 90_penguins.patch to new/debian/patches
- no, it shouldn't go to new. it should go to /whereever/you/unpacked/the/source/udev-093 (the original source dir, not the temporary working copies in /tmp)
08:25 pitti Pretty much work, isn't it? Since this happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose. 08:25 pitti that's the script you downloaded as part of the preparations 08:26 pitti so, please remove debian/patches/90_penguins.patch again 08:26 pitti yes, I know it was hard to do, but for the sake of demonstration :) 08:26 pitti so, with Pitti's Hackish Script (tm): 08:26 pitti dsrc-new-patch 90_penguins.patch 08:26 pitti sed -i 's/Linux/Penguin/g' README 08:26 pitti <press Control-D to leave the subshell> 08:27 pitti that looks slightly better, doesn't it? 08:27 pitti but I had to torture you with the close-to-the-metal method for the sake of understanding.
[at2000] in which dir to run the script?
- in the root of the unpacked source tree to conform with my previous example, /whereever/you/unpacked/the/source/udev-093
[at2000] then it should be "../dsrc-new-patch"
- oh, right, you have to specify the path to the scrpit. I have it in my ~/bin, conveniently
[jonh_wendell] is this script the same of cdb... showed yesterday?
- no, not at all. I'll come to that
[pschulz01] Hmm.. where do I find 'debclean'?
- sudo apt-get install devscripts. sorry; forgot too mention that
[jonh_wendell] before run dsrc-new-patch 90_penguins.patch we have to clean dir, right?
- if you just unpacked the tree, it shuoldn't be necessary
08:31 pitti ok, we need to go on 08:31 pitti let's stay at udev a bit and do the second example 08:31 pitti I promise, this will be the last complicated issue in this lesson 08:31 pitti but it might teach you to never ever do a package without a proper patch system :) 08:31 pitti dsrc-new-patch is currently too dumb to edit existing patches, or to put patches somewhere else than the top of the patch stack. If you need this, then you need to do the manual approach. Assume we want to edit 50-result-whitespace.patch.
[jonh_wendell] as i've run debian/rules patch with 90-penguim, i have to clean it
- ah, right
08:32 pitti This already existing patch 50-result-whitespace.patch does not depend on the previous patches, so it's a bit easier that way, too: 08:32 pitti cd /whereever/you/unpacked/the/source/udev-093 08:32 pitti cp -a . /tmp/old 08:32 pitti pushd /tmp/old 08:32 pitti cp -a . /tmp/new; cd ../new 08:32 pitti nothing new so far 08:32 pitti but note that we did not apply any patch (since we assume that 50-result-whitespace.patch does not depend on its predecessors) 08:33 pitti patch -Nlp1 < debian/patches/50-result-whitespace.patch 08:33 pitti above will apply the current version of the patch, so that you can edit it 08:33 pitti now we have the *original* change in /tmp/new, which we now want to modify 08:33 pitti so, let's do another braindead change: 08:33 pitti sed -i '1 s/$/***** HELLO WORLD ****/' udev_selinux.c 08:34 pitti or any other change you quickly want to do in your favourite editor, of course 08:34 pitti so, again we now have the original version in /tmp/old, and our old and new patch in /tmp/new 08:34 pitti so let's do the patch: 08:34 pitti cd .. 08:34 pitti diff -Nurp old new > 50-result-whitespace.patch 08:34 pitti popd 08:34 pitti mv /tmp/50-result-whitespace.patch debian/patches 08:34 pitti rm -rf /tmp/old /tmp/new 08:34 pitti Now debian/patches/50-result-whitespace.patch contains both the original change and our modification of the first line of the source file. 08:35 pitti but promised, from now on it will get really easy :) 08:35 pitti any questions? is the issue of patch dependencies clear to everyone?
[jonh_wendell] how do i know if a patch depends on others or not?
- in general you should assume that each patch depends on the previous one. they *don't* if they patch a file that hasn't been patched before or they patch it on a completely different position. e.g., if debian/patches/1_foo.patch changes foo.c, line 5 and debian/patches/2_bar.patch changes foo.c at line 6 then patch 2 definitively depends on patch 1. a (non-solid) rule of thumb is: if you can apply the patch alone on the unmodified source tree, it's independent
[at2000] is unified patch format better eliminating patch dependency?
- no, not at all. unidiff vs. context diff is purely a matter of readability. patch dependency is an inherent property of patches modifying the same thing not a question of a particular patch format
08:38 pitti Since this is so hideously complicated, patch systems were invented to aid you with that. 08:38 pitti Let's look at the most popular ones now (they are sufficient to allow you to patch about 90% of the archive's source packages; for the rest you have to resort to the manual approach above). 08:39 pitti == pmount: cdbs with simple-patchsys == 08:39 pitti cdbs' simple-patchsys.mk module matches its name, it has no bells and whistles whatsoever. However, it is pretty popular since it is sufficient for most tasks, and long ago I wrote a script 'cdbs-edit-patch' which most people can live with pretty well. This script is contained in the normal cdbs package. 08:39 pitti You just supply the name of a patch to the script, and depending on whether it already exists or not, it will create a new patch or edit an existing one. 08:39 pitti simple-patchsys is the cdbs module that 90% of all cdbs-using source packages will use 08:39 pitti BTW, "cdbs-edit-patch" is slightly misleading, since it actually only applies to simple-patchsys.mk. You can also use other cdbs patch system plugins, such as dpatch or quilt. 08:40 pitti everyone please look in debian/patches, debian/rules of pmount to get a feeling how it looks like 08:40 pitti so, let's mess up pmount a bit 08:40 pitti and add a new patch 08:40 pitti cd /whereever/you/unpacked/the/source/pmount-0.9.13 08:40 pitti cdbs-edit-patch 03-simple-readme.patch 08:40 pitti echo 'This should document pmount' > README 08:40 pitti <press Control-D to leave the subshell> 08:40 pitti easy, isn't it? 08:41 pitti check debian/patches/03-simple-readme.patch for the result 08:41 pitti this will take care of applying all patches that need to be applied, can change patches in the middle of the stack, and also create new ones 08:41 pitti Editing an already existing patch works exactly the same way, so I won't give a demo 08:42 pitti unfortunately it only really works for packages that use cdbs 08:43 pitti which a good part of Gnome-related packages do
[jonh_wendell] how do i know if a package uses cdbs?
- debian/rules includes files from /usr/share/cdbs and build-depends on cdbs
[pschulz01] What does cdbs stand for?
- common debian build system. it's a very modular abstraction of boilerplate build rules, patch systems, etc.
[at2000] when I cdbs-edit-patch the same diff, it results in 2 diffs concatenated in the same file?
- if you edit two different files, yes. in general, it will merge the old and new patch
[jonh_wendell] when we make a patch, should we add entries in debian/changelog file too?
- if you intend to publish this, yes, by all means
08:46 pitti == ed: dpatch == 08:46 pitti dpatch is a pretty robust and proven patch system which also ships a script 'dpatch-edit-patch' 08:46 pitti The two most important things you should be aware of: 08:46 pitti * dpatch does not apply debian/patches/*, but instead applies all patches mentioned in debian/patches/00list, in the mentioned order. That means that you do not have to rely on asciibetical ordering of the patches and can easily disable patches, but you have to make sure to not forget to update 00list if you add a new patch. 08:46 pitti (forgetting to update 00list is a common cause of followup uploads) 08:46 pitti * dpatch patches are actually scripts that are executed, not just patches fed to 'patch'. That means you can also do fancy things like calling autoconf or using sed in a dpatch if you want. 08:47 pitti using dpatch for non-native patches is rare, and normally you do not need to worry about how a .dpatch file looks like 08:47 pitti but I think it's important to mention it 08:47 pitti so if you ever want to replace *all* instances of Debian with Ubuntu in all files, write a dpatch with a small shell script that uses sed 08:47 pitti instead of doing a 300 KB static patch which won't apply to the next version anyway 08:47 pitti The manpage is very good and has examples, too, so I will only give an example here: 08:47 pitti This will edit an already existing patch and take care that all previous patches are applied in order: 08:48 pitti cd /whereever/you/unpacked/the/source/ed-0.2 08:48 pitti dpatch-edit-patch 05_ed.1-warning-fix 08:48 pitti <edit stuff, press Ctrl+D> 08:48 pitti so that's exactly like cdbs-edit-patch 08:48 pitti well, dpatch-edit-patch even has shell completion (!) 08:48 pitti s/shell/tab/ 08:48 pitti ok, now we edited a patch, that's pretty easy, right? 08:49 pitti now let's create a new one; this is different from cdbs-e-p, due to the explicit '00list' file 08:49 pitti dpatch-edit-patch foo.dpatch 06_testsuite-Makefile.dpatch 08:49 pitti <edit stuff, press Ctrl+D> 08:49 pitti echo foo.dpatch >> debian/patches/00list 08:49 pitti ^ this will append 'foo.dpatch' to debian/patches/00list 08:50 pitti This will create a new patch foo.dpatch relative to the already existing 06_testsuite-Makefile.dpatch. If your patch is very confined and does not depend on other patches, you can leave out the second argument. 08:50 pitti alright? 08:50 pitti there is actually yet another patch system I wanted to introduce (quilt), but I'll skip this for time reasons 08:51 Seveas you could continue after simon finished talking bugs 08:51 pitti maybe we can get to that on Thursday, or I'll explain it separately 08:51 pitti and there is existing documentation anyway 08:51 pitti so, just some more theoretic words, which you can use to calm down your fingers :) 08:51 pitti === A glimpse into the future === 08:52 pitti As you saw, Debian source packages do not have any requirements wrt. structure, patch systems, etc., other source package systems like SRPM are much stricter wrt that. This of course means more flexibility, but also much more learning overhead. 08:52 pitti As a member of the security team I can tell tales of the pain of a gazillion different source package layouts... :) 08:52 pitti Therefore some clever people sat together the other day to propose a new design which would both give us a new and unified source package and patch system that uses bzr (with a quilt-like workflow). This would also integrate packages and patches much better into Launchpad and revision control in general. 08:52 pitti Please take a look at https://wiki.ubuntu.com/NoMoreSourcePackages if you are interested in this. 08:52 pitti ah, and of course: 08:53 pitti === Notes === 08:53 pitti (apart from the obligatory channel log from fabbione) 08:53 pitti There is also a wiki page https://wiki.ubuntu.com/MOTU/School/PatchingSources which provides most of above information in a more convenient format. However, it might be slightly out of date (it's from dapper times). Feel free to update the page and and add missing bits.
[jonh_wendell] we should add manually foo.dpatch in 00list, right?
right. that's what I did with "echo foo.dpatch >> debian/patches/00list". This will append 'foo.dpatch' to debian/patches/00list
[at2000] what are pros and cons of quilt?
- hm, matter of taste, mainly. the patching workflow is not really different
[at2000] so why not just stick to cdbs, what most packages use?
- we cannot prescribe Debian to change source packages to our wishes. also, cdbs is not the unanimous solution for Ubuntu devs either. some people really hate it
[ajmitch] has there been any more progress on the NoMoreSourcePackages spec?
- bzr-wise, yes, but there's no LP support yet and it won't come in the near future, there are much more pressing TODOs for soyuz ATM
08:56 pitti I know it was hard stuff, I hope I didn't kill too many listeners 08:56 pitti any feedback for me? 08:57 pitti (but patching source packages is not easy...)
[jonh_wendell] any talk about debdiff? is it a diff of a diff?
- 'debdiff' is a tool that gives you a diff between two complete source packages. so, if you debdiff two source packages which include patches, then yes, you'll get diffs of diffs
09:00 pitti would it have been better to first explain the tools and then do the manual approach? === pitti will try this on Thursday