## page was renamed from MeetingLogs/OpenWeek PatchPackaging
== Ubuntu Open Week - Patching Packages - Tue, Nov 28, 2006 ==
see also [[MeetingLogs/openweekedgy/PatchPackaging2|Thursday Session]].
{{{
08:00	pitti	WELCOME TO MY CLASS
08:01	pitti	I have a rather technical presentation now, for everyone who wants to modify anything in Ubuntu
08:01	pitti	just as a quick overview, who is here to learn about patching?
08:02	pitti	great!
08:02	pitti	ok, time's limited, let's start
08:02	pitti	if anyone has any question, or I'm totally uncomprehensible (sorry, I'm German), please do not hesitate to interrupt and ask *immediately*
08:02	pitti	Also, don't bother trying to take notes, we'll sort that out at the end. You can fully concentrate on the discussion and examples.
08:02	pitti	Let's begin with a little bit of history:
08:02	pitti	== Why use separate patches ==
08:02	pitti	In earlier times, people just applied patches inline (i. e. directly in the source code tree). However, this makes it very hard to extract patches later to modify them, send them upstream, etc. Also this means that new upstream versions are a pain, since they generate a lot of rejections when applying the package diff.gz to them.
08:02	pitti	With split-out patches it is much easier to send them upstream, keep track of them, develop them, etc., since you always see which changes belong together.
08:02	pitti	The ideal state is an unmodified tarball from upstream, plus clean and separate patches, plus the packaging bits in debian/. That means that lsdiff <sourcepackage>.diff.gz only contains debian/.
08:02	pitti	The first attempts to split-out patches were pretty trivial: storing patches in debian/patches/, and adding some patch/patch -R snippets to debian/rules. This worked for small patches, but provided no tools for editing these patches, updating them for new upstream versions, etc.
08:02	pitti	Thus several standard patch systems were created which are easy to deploy and provide tools for patch juggling and editing.
08:04	pitti	What I would like to do now is to introduce the most common patch systems and show some hands-on demo how to add a new patch and how to edit one. For this, I will point at a source package from the current dapper archive, quickly explain the patch system, and show how to apply some (braindead) modifications to it. I recommend you to do the same steps in a terminal, so that you get a feeling for the process and can immediately ask questions.
08:04	pitti	everyone you fine with this approach?
08:04	pitti	erm, of course I mean 'from the edgy archive', not dapper
08:04	pitti	yeah, I don't want it to be entirely theoretical :)
08:05	pitti	If you want to try the stuff yourself, please do the following commands (on edgy) as preparation:
08:05	pitti	  sudo apt-get install dpatch cdbs quilt patchutils
08:05	pitti	  apt-get source cron udev pmount gnome-volume-manager ed xterm
08:05	pitti	  wget http://people.ubuntu.com/~pitti/scripts/dsrc-new-patch
08:05	pitti	  chmod 755 dsrc-new-patch
08:05	pitti	I deliberately picked the smallest packages I could find
08:08	pitti	ok, everyone set with the intro and getting source packages?
08:09	pitti	nevermind, for the first bit we don't yet need the packages
08:09	pitti	== cron: inline patches ==
08:09	pitti	No patch system at all, nothing much to say about this.  You directly edit the files in the source tree. This is convenient for a simple and quick change, but will bite back for new upstream versions (see above) and is inconvenient for submitting patches upstream, or reviewing for merges.
08:09	pitti	if you do 'lsdiff <package>.diff.gz' and you see changes which are not in debian/, then you probably have such a package
08:09	pitti	(some KDE packages have autoconf stuff directly in the diff.gz, but that is ok)
08:09	pitti	so, I think I do not need to say anything else about cron, unless someone has a question
08:10	pitti	ok, no question here
08:10	pitti	so, people, get a grip, now it gets heavy
08:11	pitti	== udev: separate patches, but no patch system ==
08:11	pitti	This case is the most complicated one since you have to do all the hard work manually. In order to make you understand what a patch system does, and to give you a fallback method that will *always* work with any patch system, I handle this first.
08:11	pitti	The good news is that you will seldomly be required to actually do this procedure, I'll come to this again.
08:11	pitti	The general approach is:
08:11	pitti	1. copy the clean source tree to a temporary directory /tmp/old
08:11	pitti	2. apply all patches up to the one you want to edit; if you want to create a new patch, apply all existing ones (this is necessary since in general patches depend on previous patches)
08:11	pitti	if you want, you can use debian/rules for this: remove the patches that come *after* the one you want to edit, and call 'debian/rules patch'. The actual name for the patch target varies, I have seen the following ones so far: patch setup apply-patches unpack patch-stamp. You have to look in debian/rules how it is called.
08:11	pitti	3. copy the whole source tree again: cp -a /tmp/old /tmp/new
08:11	pitti	4. go into /tmp/new, do your modifications
08:11	pitti	5. go back into /tmp generate the patch with
08:11	pitti	  diff -Nurp old new > mypatchname.patch
08:11	pitti	and move the patch to .../debian/patches/mypatchname.patch
08:11	pitti	^ that's the theory, example will follow shortly (hands-on training)
}}}
''[jonh_wendell]	pitti, i do a lsdiff <package>.diff and got nothing''

 that won't work in that form - lsdiff -z package.diff.gz. -z is for 'accept gzipped files', source packages have a diff.gz, not a .diff

{{{
08:12	pitti	these steps are the general method how any patch is handled
08:14	pitti	in general we want the following diff options:
08:14	pitti	-N -> include new files
08:14	pitti	-u -> unified patches (context diffs are ugly)
08:14	pitti	-r -> recursive
08:14	pitti	-p -> bonus, you can see the name of the affected function in the patch
08:14	pitti	does anyone have a question about the principle method?
08:14	pitti	(I know it's heavy, sorry, that's live; promised, it'll get easier)
}}}
''[jonh_wendell]	pitti, about cron, are you showing a wrong way to package?''

 well, it's not exactly 'wrong' as long as the package is not heavily modified, or Debian/Ubuntu is essentially the upstream, it's fine, but having all patches in a single big lump makes it next to impossible to send stuff upstream, or tell which bit belongs to what. also, the tendency is that packages which have 'always' been there, have more 'archaic' build systems and no patch system where as most recently packaged stuff uses a standard packaging system with a standard patch system, now that they are invented

''[mwicki]	does anybody know how to install a debianpaket whitout root-permission, is it possible to install a debian-paket in a directory??''

 [Jon] you can unpack it using dpkg-deb -x <package> <directory> as any user who can write to <directory>. mwicki: be warned, though, that doesn't run the pre or post install scripts

 do you need to do that for patching? it most probably won't work. if it's a simple program then you could then run the binaries in it
{{{
08:17	pitti	ok, let's go back to patching udev - an EXAMPLE! \o/
08:17	pitti	open a shell, ready your fingers :)
08:17	pitti	udev example 1, let's create a new patch 90_penguins.patch:
08:17	pitti	  cd /whereever/you/unpacked/the/source/udev-093
08:17	pitti	  cp -a . /tmp/old
08:17	pitti	  pushd /tmp/old
08:17	pitti	  debian/rules patch
08:17	pitti	  cp -a . /tmp/new; cd ../new
08:17	pitti	that's the preparatory part, now we have a directory we can hack in
08:18	pitti	let's do a braindead modification
08:18	pitti	  sed -i 's/Linux/Penguin/g' README
08:18	pitti	^ of course you can also use a normal editor
08:18	pitti	I just use sed so that you can copy&paste the commands without any interactive stuff
08:19	pitti	and now we create a patch between the reference and our new tree
08:19	pitti	  cd ..
08:19	pitti	  diff -Nurp old new > 90_penguins.patch
08:19	pitti	  popd
08:19	pitti	  rm -rf /tmp/old /tmp/new
08:19	pitti	  mv /tmp/90_penguins.patch debian/patches
08:19	pitti	(Ignore the 'recursive directory loop' warnings)
}}}
''[jonh_wendell]	what does pushd do?''

 'pushd foo' is like 'cd foo', but remembers the last dir and pushes it onto a stack. if you then do 'popd', it will return to the directory you have been in when you did 'pushd'

 ''[StaCk]	isn't it like cd - ?''

 no, that will go only to the previous dir. you can stack multiple pushd's on top of each other, and allow 'cd' in between
{{{
08:21	pitti	Now take a look at your shiny new debian/patches/90_penguins.patch.
08:21	pitti	*uff* :)
08:21	pitti	after that, if you do 'debian/rules patch', you'll see that the patch applies cleanly
08:21	pitti	
08:22	pitti	so, obviously that's not the end of the wisdom, but if you do these steps a couple of times, you should get a feeling for how to create the most complicated patch conceivable
08:22	pitti	so this procedure is the life safer if anything else fails
08:22	pitti	questions so far?
}}}
''[pschulz01]	pitti: cd new?''

 'cp -a . /tmp/new; cd ../new' We applied all previous patches in /tmp/old, then copied /tmp/old to /tmp/new, and hacked our stuff in /tmp/new


''[dieterd]	result is then a new cd with included patches?''

 the result after all of these steps is a single new file debian/patches/90_penguins.patch. the commands above removed the temporary /tmp/old and /tmp/new trees

''[pschulz01]	Need to copy 90_penguins.patch to new/debian/patches''

 no, it shouldn't go to new. it should go to /whereever/you/unpacked/the/source/udev-093 (the original source dir, not the temporary working copies in /tmp)

{{{
08:25	pitti	Pretty much work, isn't it? Since this happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose.
08:25	pitti	that's the script you downloaded as part of the preparations
08:26	pitti	so, please remove debian/patches/90_penguins.patch again
08:26	pitti	yes, I know it was hard to do, but for the sake of demonstration :)
08:26	pitti	so, with Pitti's Hackish Script (tm):
08:26	pitti	  dsrc-new-patch 90_penguins.patch
08:26	pitti	  sed -i 's/Linux/Penguin/g' README
08:26	pitti	  <press Control-D to leave the subshell>
08:27	pitti	that looks slightly better, doesn't it?
08:27	pitti	but I had to torture you with the close-to-the-metal method for the sake of understanding.
}}}
''[at2000]	in which dir to run the script?''

 in the root of the unpacked source tree to conform with my previous example, /whereever/you/unpacked/the/source/udev-093

 ''[at2000]	then it should be "../dsrc-new-patch"''

  oh, right, you have to specify the path to the scrpit. I have it in my ~/bin, conveniently

''[jonh_wendell]	is this script the same of cdb... showed yesterday?''

 no, not at all.  I'll come to that

''[pschulz01]	Hmm.. where do I find 'debclean'?''

 sudo apt-get install devscripts. sorry; forgot too mention that

''[jonh_wendell]	before run  dsrc-new-patch 90_penguins.patch  we have to clean dir, right?''

 if you just unpacked the tree, it shuoldn't be necessary
{{{
08:31	pitti	ok, we need to go on
08:31	pitti	let's stay at udev a bit and do the second example
08:31	pitti	I promise, this will be the last complicated issue in this lesson
08:31	pitti	but it might teach you to never ever do a package without a proper patch system :)
08:31	pitti	dsrc-new-patch is currently too dumb to edit existing patches, or to put patches somewhere else than the top of the patch stack. If you need this, then you need to do the manual approach. Assume we want to edit 50-result-whitespace.patch.
}}}
''[jonh_wendell]	as i've run debian/rules patch with 90-penguim, i have to clean it''

 ah, right
{{{
08:32	pitti	This already existing patch 50-result-whitespace.patch does not depend on the previous patches, so it's a bit easier that way, too:
08:32	pitti	  cd /whereever/you/unpacked/the/source/udev-093
08:32	pitti	  cp -a . /tmp/old
08:32	pitti	  pushd /tmp/old
08:32	pitti	  cp -a . /tmp/new; cd ../new
08:32	pitti	nothing new so far
08:32	pitti	but note that we did not apply any patch (since we assume that 50-result-whitespace.patch does not depend on its predecessors)
08:33	pitti	  patch -Nlp1 < debian/patches/50-result-whitespace.patch
08:33	pitti	above will apply the current version of the patch, so that you can edit it
08:33	pitti	now we have the *original* change in /tmp/new, which we now want to modify
08:33	pitti	so, let's do another braindead change:
08:33	pitti	  sed -i '1 s/$/***** HELLO WORLD ****/' udev_selinux.c
08:34	pitti	or any other change you quickly want to do in your favourite editor, of course
08:34	pitti	so, again we now have the original version in /tmp/old, and our old and new patch in /tmp/new
08:34	pitti	so let's do the patch:
08:34	pitti	  cd ..
08:34	pitti	  diff -Nurp old new > 50-result-whitespace.patch
08:34	pitti	  popd
08:34	pitti	  mv /tmp/50-result-whitespace.patch debian/patches
08:34	pitti	  rm -rf /tmp/old /tmp/new
08:34	pitti	Now debian/patches/50-result-whitespace.patch contains both the original change and our modification of the first line of the source file.
08:35	pitti	but promised, from now on it will get really easy :)
08:35	pitti	any questions? is the issue of patch dependencies clear to everyone?
}}}
''[jonh_wendell]	how do i know if a patch depends on others or not?''

 in general you should assume that each patch depends on the previous one. they *don't* if they patch a file that hasn't been patched before or they patch it on a completely different position. e.g., if debian/patches/1_foo.patch changes foo.c, line 5 and debian/patches/2_bar.patch changes foo.c at line 6 then patch 2 definitively depends on patch 1. a (non-solid) rule of thumb is: if you can apply the patch alone on the unmodified source tree, it's independent

''[at2000]	is unified patch format better eliminating patch dependency?''

 no, not at all. unidiff vs. context diff is purely a matter of readability. patch dependency is an inherent property of patches modifying the same thing	not a question of a particular patch format

{{{
08:38	pitti	Since this is so hideously complicated, patch systems were invented to aid you with that.
08:38	pitti	Let's look at the most popular ones now (they are sufficient to allow you to patch about 90% of the archive's source packages; for the rest you have to resort to the manual approach above).
08:39	pitti	== pmount: cdbs with simple-patchsys ==
08:39	pitti	cdbs' simple-patchsys.mk module matches its name, it has no bells and whistles whatsoever. However, it is pretty popular since it is sufficient for most tasks, and long ago I wrote a script 'cdbs-edit-patch' which most people can live with pretty well. This script is contained in the normal cdbs package.
08:39	pitti	You just supply the name of a patch to the script, and depending on whether it already exists or not, it will create a new patch or edit an existing one.
08:39	pitti	simple-patchsys is the cdbs module that 90% of all cdbs-using source packages will use
08:39	pitti	BTW, "cdbs-edit-patch" is slightly misleading, since it actually only applies to simple-patchsys.mk. You can also use other cdbs patch system plugins, such as dpatch or quilt.
08:40	pitti	everyone please look in debian/patches, debian/rules of pmount to get a feeling how it looks like
08:40	pitti	so, let's mess up pmount a bit
08:40	pitti	and add a new patch
08:40	pitti	  cd /whereever/you/unpacked/the/source/pmount-0.9.13
08:40	pitti	  cdbs-edit-patch 03-simple-readme.patch
08:40	pitti	  echo 'This should document pmount' > README
08:40	pitti	  <press Control-D to leave the subshell>
08:40	pitti	easy, isn't it?
08:41	pitti	check debian/patches/03-simple-readme.patch for the result
08:41	pitti	this will take care of applying all patches that need to be applied, can change patches in the middle of the stack, and also create new ones
08:41	pitti	Editing an already existing patch works exactly the same way, so I won't give a demo
08:42	pitti	unfortunately it only really works for packages that use cdbs
08:43	pitti	which a good part of Gnome-related packages do
}}}
''[jonh_wendell] how do i know if a package uses cdbs?''

 debian/rules includes files from /usr/share/cdbs and build-depends on cdbs

''[pschulz01]	What does cdbs stand for?''

 common debian build system. it's a very modular abstraction of boilerplate build rules, patch systems, etc.

''[at2000]	when I cdbs-edit-patch the same diff, it results in 2 diffs concatenated in the same file?''

 if you edit two different files, yes. in general, it will merge the old and new patch

''[jonh_wendell]	when we make a patch, should we add entries in debian/changelog file too?''

 if you intend to publish this, yes, by all means
{{{
08:46	pitti	== ed: dpatch ==
08:46	pitti	dpatch is a pretty robust and proven patch system which also ships a script 'dpatch-edit-patch'
08:46	pitti	The two most important things you should be aware of:
08:46	pitti	* dpatch does not apply debian/patches/*, but instead applies all patches mentioned in debian/patches/00list, in the mentioned order. That means that you do not have to rely on asciibetical ordering of the patches and can easily disable patches, but you have to make sure to not forget to update 00list if you add a new patch.
08:46	pitti	(forgetting to update 00list is a common cause of followup uploads)
08:46	pitti	* dpatch patches are actually scripts that are executed, not just patches fed to 'patch'. That means you can also do fancy things like calling autoconf or using sed in a dpatch if you want.
08:47	pitti	using dpatch for non-native patches is rare, and normally you do not need to worry about how a .dpatch file looks like
08:47	pitti	but I think it's important to mention it
08:47	pitti	so if you ever want to replace *all* instances of Debian with Ubuntu in all files, write a dpatch with a small shell script that uses sed
08:47	pitti	instead of doing a 300 KB static patch which won't apply to the next version anyway
08:47	pitti	The manpage is very good and has examples, too, so I will only give an example here:
08:47	pitti	This will edit an already existing patch and take care that all previous patches are applied in order:
08:48	pitti	  cd /whereever/you/unpacked/the/source/ed-0.2
08:48	pitti	  dpatch-edit-patch 05_ed.1-warning-fix
08:48	pitti	  <edit stuff, press Ctrl+D>
08:48	pitti	so that's exactly like cdbs-edit-patch
08:48	pitti	well, dpatch-edit-patch even has shell completion (!)
08:48	pitti	s/shell/tab/
08:48	pitti	ok, now we edited a patch, that's pretty easy, right?
08:49	pitti	now let's create a new one; this is different from cdbs-e-p, due to the explicit '00list' file
08:49	pitti	  dpatch-edit-patch foo.dpatch 06_testsuite-Makefile.dpatch
08:49	pitti	  <edit stuff, press Ctrl+D>
08:49	pitti	  echo foo.dpatch >> debian/patches/00list
08:49	pitti	^ this will append 'foo.dpatch' to debian/patches/00list
08:50	pitti	This will create a new patch foo.dpatch relative to the already existing 06_testsuite-Makefile.dpatch. If your patch is very confined and does not depend on other patches, you can leave out the second argument.
08:50	pitti	alright?
08:50	pitti	there is actually yet another patch system I wanted to introduce (quilt), but I'll skip this for time reasons
08:51	Seveas	you could continue after simon finished talking bugs
08:51	pitti	maybe we can get to that on Thursday, or I'll explain it separately
08:51	pitti	and there is existing documentation anyway
08:51	pitti	so, just some more theoretic words, which you can use to calm down your fingers :)
08:51	pitti	=== A glimpse into the future ===
08:52	pitti	As you saw, Debian source packages do not have any requirements wrt. structure, patch systems, etc., other source package systems like SRPM are much stricter wrt that. This of course means more flexibility, but also much more learning overhead.
08:52	pitti	As a member of the security team I can tell tales of the pain of a gazillion different source package layouts... :)
08:52	pitti	Therefore some clever people sat together the other day to propose a new design which would both give us a new and unified source package and patch system that uses bzr (with a quilt-like workflow). This would also integrate packages and patches much better into Launchpad and revision control in general.
08:52	pitti	Please take a look at https://wiki.ubuntu.com/NoMoreSourcePackages if you are interested in this.
08:52	pitti	ah, and of course:
08:53	pitti	=== Notes ===
08:53	pitti	(apart from the obligatory channel log from fabbione)
08:53	pitti	There is also a wiki page https://wiki.ubuntu.com/MOTU/School/PatchingSources which provides most of above information in a more convenient format. However, it might be slightly out of date (it's from dapper times). Feel free to update the page and and add missing bits.
}}}
''[jonh_wendell]	we should add manually foo.dpatch in 00list, right?''

 right. that's what I did with  "echo foo.dpatch >> debian/patches/00list". This will append 'foo.dpatch' to debian/patches/00list

''[at2000]	what are pros and cons of quilt?''

 hm, matter of taste, mainly. the patching workflow is not really different

''[at2000]	so why not just stick to cdbs, what most packages use?''

 we cannot prescribe Debian to change source packages to our wishes. also, cdbs is not the unanimous solution for Ubuntu devs either. some people really hate it

''[ajmitch]	has there been any more progress on the NoMoreSourcePackages spec?''

 bzr-wise, yes, but there's no LP support yet and it won't come in the near future, there are much more pressing TODOs for soyuz ATM
{{{
08:56	pitti	I know it was hard stuff, I hope I didn't kill too many listeners
08:56	pitti	any feedback for me?
08:57	pitti	(but patching source packages is not easy...)
}}}
''[jonh_wendell]	any talk about debdiff? is it a diff of a diff?''

 'debdiff' is a tool that gives you a diff between two complete source packages.  so, if you debdiff two source packages which include patches, then yes, you'll get diffs of diffs
{{{
09:00	pitti	would it have been better to first explain the tools and then do the manual approach?
=== pitti will try this on Thursday}}}