## page was renamed from MeetingLogs/OpenWeek PatchPackaging2
== Ubuntu Open Week - Patching Packages - Thu, Nov 30, 2006 ==
see also [[MeetingLogs/openweekedgy/PatchPackaging|Tuesday Session]].
{{{
06:00	sfllaw	Patching Packages with Martin Pitt (pitti)
06:01	pitti	so, who are the Happy Patch People around here? :0
=== jrib raises hand
06:01	pitti	just a warning, today's talk will be 95% the same as the one I held on Tuesday
=== jorgp raises hand, Im here
06:02	pitti	jrib, jorgp: welcome
06:02	ailean	me :)
06:02	jrib	hmm do you think you can get a chance to cover quilt?
06:02	pitti	ah, seems that most of the folks already attended Tuesday
06:02	pitti	jrib: yes, if time permits
=== pitti clears his throat
06:02	pitti	if anyone has any question, or I'm totally uncomprehensible (sorry, I'm German), please do not hesitate to interrupt and ask *immediately*
06:02	pitti	Also, don't bother trying to take notes, we'll sort that out at the end. You can fully concentrate on the discussion and examples.
06:02	pitti	Let's begin with a little bit of history:
06:03	pitti	oh, I will usually paste a snippet and then give you some time to catch up and ask, ok?
06:03	pitti	== Why use separate patches ==
06:03	pitti	In earlier times, people just applied patches inline (i. e. directly in the source code tree). However, this makes it very hard to extract patches later to modify them, send them upstream, etc. Also this means that new upstream versions are a pain, since they generate a lot of rejections when applying the package diff.gz to them.
06:03	pitti	With split-out patches it is much easier to send them upstream, keep track of them, develop them, etc., since you always see which changes belong together.
06:03	pitti	The ideal state is an unmodified tarball from upstream, plus clean and separate patches, plus the packaging bits in debian/. That means that lsdiff <sourcepackage>.diff.gz only contains debian/.
06:04	pitti	The first attempts to split-out patches were pretty trivial: storing patches in debian/patches/, and adding some patch/patch -R snippets to debian/rules. This worked for small patches, but provided no tools for editing these patches, updating them for new upstream versions, etc.
06:04	pitti	Thus several standard patch systems were created which are easy to deploy and provide tools for patch juggling and editing.
06:04	pitti	any general questions at this point?
06:05	pitti	What I would like to do now is to introduce the most common patch systems and show some hands-on demo how to add a new patch and how to edit one. For this, I will point at a source package from the current dapper archive, quickly explain the patch system, and show how to apply some (braindead) modifications to it. I recommend you to do the same steps in a terminal, so that you get a feeling for the process and can immediately ask questions.
06:05	pitti	everyone you fine with this approach?
06:05	pitti	If you want to try the stuff yourself, please do the following commands (on edgy) as preparation:
06:05	pitti	  sudo apt-get install dpatch cdbs quilt patchutils devscripts
06:05	pitti	  apt-get source cron udev pmount gnome-volume-manager ed xterm
06:05	pitti	  wget http://people.ubuntu.com/~pitti/scripts/dsrc-new-patch
06:05	pitti	  chmod 755 dsrc-new-patch
06:05	pitti	I deliberately picked the smallest packages I could find
=== pitti waits a bit for people to do the preparations
06:08	pitti	== cron: inline patches ==
06:08	pitti	No patch system at all, nothing much to say about this.  You directly edit the files in the source tree. This is convenient for a simple and quick change, but will bite back for new upstream versions (see above) and is inconvenient for submitting patches upstream, or reviewing for merges.
06:08	pitti	if you do 'lsdiff <package>.diff.gz' and you see changes which are not in debian/, then you probably have such a package
06:08	pitti	(some KDE packages have autoconf stuff directly in the diff.gz, but that is ok)
06:08	pitti	so, I think I do not need to say anything else about cron, unless someone has a question
06:08	pitti	this style of packages can be mainly found in (1) packages which are around for ages (like cron) or packages where Debian/Ubuntu is upstream
06:09	pitti	or when the maintainer simply thinks he won't need a patch system
06:09	pitti	:)
06:09	pitti	ok, then let's dive in
06:09	pitti	== udev: separate patches, but no patch system ==
06:09	pitti	This case is the most complicated one since you have to do all the hard work manually. In order to make you understand what a patch system does, and to give you a fallback method that will *always* work with any patch system, I handle this first.
06:09	pitti	The good news is that you will seldomly be required to actually do this procedure, since for many packages there are nice tools which make things a charm.
06:09	pitti	The bad news is that it may seem utterly complicated for people who never did it before, but I would like you to understand what's actually going on behind the curtains of the tools.
06:10	pitti	since it is your live safer
06:10	pitti	life, even
06:10	pitti	So please do not desperate if you do not fully understand it at first; there's written documentation and you can always take your time to grok it.
06:10	pitti	The general approach is:
06:10	pitti	oh, please ask questions right here, as I asked for
06:10	pitti	I like to treat them synchronously, since this is a hands-on tutorial
}}}

''<jrib> we need  lsdiff -z <package>.diff.gz  right?''

 right, that will show what the diff.gz touches. ideally this is just debian/

''<ailean> I'm a complete beginner to all of this . . . I'm unsure about what cron is, and what .diff.gz is. Should I consult something else before coming back to read these logs?''

 I'm afraid you might be overwhelmed by this stuff. I do assume that you have basic familiarity about what a source pacakge is and how it looks like. this is intended to be a tutorial for people who want to hack on packages,  but of course feel free to listen. cron is 'just a random package' for our purposes  and a diff.gz is the disto specific part on top of the original upstream tarball, with all the packaging specific bits and code changes, but due to limited time I can't be more specific here
{{{
06:13	pitti	ok, so the general approach
06:13	pitti	1. copy the clean source tree to a temporary directory /tmp/old
06:13	pitti	2. apply all patches up to the one you want to edit; if you want to create a new patch, apply all existing ones (this is necessary since in general patches depend on previous patches)
06:13	pitti	3. copy the whole source tree again: cp -a /tmp/old /tmp/new
06:13	pitti	4. go into /tmp/new, do your modifications
06:13	pitti	5. go back into /tmp and generate the patch with
06:13	pitti	  diff -Nurp old new > mypatchname.patch
06:13	pitti	6. move the newly generated patch to <original source dir>/debian/patches/mypatchname.patch
06:14	pitti	(just the theory, detailled explanation will follow shortly)
06:14	pitti	with the demo it will become clear
06:14	pitti	just an explanation of the '-Nurp' diff options:
06:14	pitti	in general we want the following diff options:
06:14	pitti	-N -> include new files
06:14	pitti	-u -> unified patches (context diffs are ugly)
06:14	pitti	-r -> recursive
06:14	pitti	-p -> bonus, you can see the name of the affected function in the patch
06:14	pitti	does anyone have a question about the principle method?
}}}

''[jrib]	when you say we need to apply  the patches up to the one we need to edit, does that mean all of the ones that start with a smaller number?''

 right.  in 95% of the cases, patches are applied in asciibetical order, and since most patches are prefixed with a number, sorted by number

''[chickn]	wait..before you go too far, i don't get how you know if a package has a 'patching system' or not..''

 ah, good question. if there is a debian/patches/ with patches, then most probably there is a patch system. there are very few braindead pacakges which apply their patches inline and additionally put them into debian/patches/ for future reference. this is just *silly*

''[chickn]	So for cron...there's no reference for 'patch' in rules''

 right, and you will see taht lsdiff -z cron*.diff.gz contains source code patches. i. e. it just applies the patches inline in the raw source tree

{{{
06:17	pitti	and it happened to the best of us that we just sticked a new patch into that and wondered why it didn't become active
06:17	pitti	you can check debian/rules and search for 'patch', that's a pretty safe method
06:18	pitti	ok, some hands-on example on udev
06:18	pitti	open a shell, ready your fingers :)
06:19	pitti	udev example 1, let's create a new patch 90_penguins.patch:
06:19	pitti	  cd /whereever/you/unpacked/the/source/udev-093
06:19	pitti	-> now we are in our original source tree where we want to add a new patch
06:20	pitti	  cp -a . /tmp/old
06:20	pitti	-> create a copy of the clean sources as reference tree
06:20	pitti	  pushd /tmp/old
06:20	pitti	-> go to /tmp/old; 'pushd' to remember the previous directory, so that we can go back conveniently
06:20	pitti	  debian/rules patch
06:20	pitti	-> apply all already existing patches; of course we could use the 'patch' program to do it manually, but since debian/rules already knows how to do it, let's use it. The actual name for the patch target varies, I have seen the following ones so far: patch, setup, apply-patches, unpack, patch-stamp. You have to look in debian/rules how it is called.
06:21	pitti	  cp -a . /tmp/new; cd ../new
06:21	pitti	-> copies our patched reference tree to our new work directory /tmp/new where we can hack in
06:21	pitti	that's the preparatory part
06:21	pitti	set so far?
06:22	pitti	let's do a braindead modification now
06:23	pitti	  sed -i 's/Linux/Penguin/g' README
06:23	pitti	-> changes the README file; of course you can use your favourite editor, but I wanted to keep my examples copy&pasteable
06:23	pitti	and now we create a patch between the reference and our new tree:
06:23	pitti	  cd ..
06:23	pitti	-> go back to /tmp, i. e. where our reference tree (old) and hacked tree (new) is located
06:23	pitti	  diff -Nurp old new > 90_penguins.patch
06:23	pitti	-> generate the patch (Ignore the 'recursive directory loop' warnings)
06:23	pitti	  popd
06:23	pitti	-> now you should be back in your original source tree (when you did the pushd)
06:24	pitti	  rm -rf /tmp/old /tmp/new
06:24	pitti	-> clean up the temporary trees
06:24	pitti	  mv /tmp/90_penguins.patch debian/patches
06:24	pitti	-> move the patch from /tmp to the source tree's patch directory, where it belongs.
06:24	pitti	*uff* :)
06:24	pitti	Now take a look at your shiny new debian/patches/90_penguins.patch.
06:24	pitti	after that, if you do 'debian/rules patch', you'll see that the patch applies cleanly; please do 'debclean' afterwards to unapply the patches and get back a pristine source tree
06:25	pitti	so, obviously that's not the end of the wisdom, but if you do these steps a couple of times, you should get a feeling for how to create the most complicated patch conceivable
06:25	pitti	so this procedure is the life safer if anything else fails
06:25	pitti	questions so far?
}}}

''[jrib]	I notice that the patches jump from 01 to 40, and sometimes only jump by 5.  What is the convention in naming the patches?''

 there's no real policy. often patches are named in ascending order, then maybe patch 05 becomes obsolete or applied upstream. sometimes, maintainers organize them in blocks.  e. g. in postgresql, upstream patches start with 00, server patches from 10 to 300, contrib patches from 300 to 400, etc. the only rule is that the asciibetical order is the correct order for applying them; they do not even need to start with a number

''[chickn]	but if you're writing patches on top of ones already applied - shouldn't your name be something that will come last?''

 right, usually you should. however, if you pull stuff from upstream, convention is that you order them first, so that the distro-specific patches stay on top.	 that makes it easier to upgrade to a new upstream version.

{{{
06:28	pitti	Pretty much work, isn't it? Since this happens pretty often, I created a very dumb helper script 'dsrc-new-patch' for this purpose.
06:29	pitti	chickn: https://wiki.ubuntu.com/MOTU/School/PatchingSources
06:29	pitti	so, please rm debian/patches/90_penguins.patch
06:30	pitti	I know, it was precious and hard to do, but now it gets easier :)
06:30	pitti	Using this script, above steps would reduce to:
06:30	pitti	  ../dsrc-new-patch 90_penguins.patch
06:30	pitti	  sed -i 's/Linux/Penguin/g' README
06:30	pitti	  <press Control-D to leave the subshell>
06:30	pitti	that looks slightly better, doesn't it? If you like the script, please put it into your ~/bin, so that it is in your $PATH
06:30	pitti	but I had to torture you with the close-to-the-metal method for the sake of understanding.
06:31	pitti	but let's stay at udev a bit and do the second example
06:31	pitti	I promise, this will be the last complicated issue in this lesson
06:31	pitti	but it might teach you to never ever do a package without a proper patch system :)
06:32	pitti	dsrc-new-patch is currently too dumb to edit existing patches, or to put patches somewhere else than the top of the patch stack. If you need this, then you need to do the manual approach.
06:32	pitti	Assume we want to edit 50-result-whitespace.patch. This patch does not depend on the previous patches, so it's a bit easier that way, too:
06:32	pitti	  cd /whereever/you/unpacked/the/source/udev-093
06:32	pitti	  cp -a . /tmp/old
06:32	pitti	  pushd /tmp/old
06:32	pitti	  cp -a . /tmp/new; cd ../new
06:32	pitti	-> nothing new so far, just created a reference and a hack tree
06:32	pitti	but note that we did not apply any patch (since we assume that 50-result-whitespace.patch does not depend on its predecessors)
06:33	pitti	everyone got that?
06:33	pitti	  patch -Nlp1 < debian/patches/50-result-whitespace.patch
06:33	pitti	above will apply the current version of the patch, so that you can edit it
06:34	pitti	we have to do that only in /tmp/new, so that our final diff call catches both the old patch and our new modifications
06:34	pitti	  sed -i '1 s/$/***** HELLO WORLD ****/' udev_selinux.c
06:34	pitti	(just some random braindead change)
06:34	pitti	now we have the updated tree in /tmp/new
06:34	pitti	so let's do the patch:
06:35	pitti	  cd ..
06:35	pitti	  diff -Nurp old new > 50-result-whitespace.patch
06:35	pitti	  popd
06:35	pitti	  mv /tmp/50-result-whitespace.patch debian/patches
06:35	pitti	  rm -rf /tmp/old /tmp/new
06:35	pitti	exactly the same commands as before
06:35	pitti	Now debian/patches/50-result-whitespace.patch contains both the original change and our modification of the first line of the source file.
06:35	pitti	but promised, from now on it will get really easy :)
06:35	pitti	any questions? is the issue of patch dependencies clear to everyone?
}}}

''[jrib]	if 50-result-whitespace.patch depended on 01-lib-udev.patch for example.  Would the procedure then just be to apply 01-lib-udev.patch to old first and then do the same thing as before?''

  you got it. if you want to stay at the safe side, just apply all patches before the one you want to edit in /tmp/old. doing that can never hurt
{{{
06:36	pitti	Since this is so hideously complicated, patch systems were invented to aid you with that. Let's look at the most popular ones now (they are sufficient to allow you to patch about 90% of the archive's source packages; for the rest you have to resort to the manual approach above).
06:37	pitti	== pmount: cdbs with simple-patchsys ==
06:37	pitti	cdbs' simple-patchsys.mk module matches its name, it has no bells and whistles whatsoever. However, it is pretty popular since it is sufficient for most tasks, and long ago I wrote a script 'cdbs-edit-patch' which most people can live with pretty well. This script is contained in the normal cdbs package.
06:37	pitti	however, of course it only works for pacakges which actually use cdbs
06:37	pitti	i. e. which have a cdbs build-dependency
06:38	pitti	and include some stuff (simple-patchsys.mk in particular) from /usr/share/cdbs
06:38	pitti	^ in debian/rules
06:38	pitti	please take a quick look at debian/rules in the pmount source
06:38	pitti	You just supply the name of a patch to the script, and depending on whether it already exists or not, it will create a new patch or edit an existing one.
06:39	pitti	this includes taking care of patch dependencies, editing patches in the middle of the stack, etc.
06:39	pitti	so, let's mess up pmount a bit
06:39	pitti	and add a new patch
06:39	pitti	  cd /whereever/you/unpacked/the/source/pmount-0.9.13
06:39	pitti	  cdbs-edit-patch 03-simple-readme.patch
06:39	pitti	  echo 'This should document pmount' > README
06:39	pitti	  <press Control-D to leave the subshell>
06:39	pitti	easy, isn't it?
06:40	pitti	ok?
06:40	jorgp	that was easy
06:40	pitti	Editing an already existing patch works exactly the same way.
06:41	pitti	so I won't give a demo
}}}
''[elliot__]	got to go now. Whats the link for the chat log?''

 https://wiki.ubuntu.com/MOTU/School/PatchingSources is the wiki page for it
{{{
06:41	pitti	BTW, "cdbs-edit-patch" is slightly misleading, since it actually only applies to simple-patchsys.mk. You can also use other cdbs patch system plugins, such as dpatch or quilt.
06:41	pitti	questions?
06:42	jrib	nope, this is a lot easier :)
06:42	pitti	I promised :)
06:42	pitti	== ed: dpatch ==
06:42	pitti	dpatch is a pretty robust and proven patch system which also ships a script 'dpatch-edit-patch'
06:42	pitti	The two most important things you should be aware of:
06:42	pitti	* dpatch does not apply debian/patches/*, but instead applies all patches mentioned in debian/patches/00list, in the mentioned order. That means that you do not have to rely on asciibetical ordering of the patches and can easily disable patches, but you have to make sure to not forget to update 00list if you add a new patch.
06:42	pitti	(forgetting to update 00list is a common cause of followup uploads)
06:43	pitti	* dpatch patches are actually scripts that are executed, not just patches fed to 'patch'. That means you can also do fancy things like calling autoconf or using sed in a dpatch if you want.
06:43	pitti	using dpatch for non-native patches is rare, and normally you do not need to worry about how a .dpatch file looks like
06:43	pitti	but I think it's important to mention it
06:43	pitti	so if you ever want to replace *all* instances of Debian with Ubuntu in all files, write a dpatch with a small shell script that uses sed
06:43	pitti	instead of doing a 300 KB static patch which won't apply to the next version anyway
06:44	pitti	The manpage is very good and has examples, too, so I will only give an example here:
06:44	pitti	This will edit an already existing patch and take care that all previous patches are applied in order:
06:44	pitti	  cd /whereever/you/unpacked/the/source/ed-0.2
06:44	pitti	  dpatch-edit-patch 05_ed.1-warning-fix
06:44	pitti	  <edit stuff, press Ctrl+D>
06:44	pitti	so that's exactly like cdbs-edit-patch
06:44	pitti	ok, now we edited a patch, that's pretty easy, right?
06:44	jorgp	yes
06:44	pitti	now let's create a new one; this is different from cdbs-edit-patch
06:44	pitti	(due to the 00list handling)
06:45	pitti	  dpatch-edit-patch foo.dpatch 06_testsuite-Makefile.dpatch
06:45	pitti	  <edit stuff, press Ctrl+D>
06:45	pitti	  echo foo.dpatch >> debian/patches/00list
06:45	pitti	This will create a new patch foo.dpatch relative to the already existing 06_testsuite-Makefile.dpatch. If your patch is very confined and does not depend on other patches, you can leave out the second argument.
06:45	pitti	please note the last action (adding your new patch at the appropriate position in the patch list)
06:46	pitti	so again, dpatch does *not* rely on any asciibetical order by default
}}}
''[jorgp]	would it still be in this scope of discussion to talk about patches that don't apply cleanly''

 of course. if you get a source package from the archive, they should apply cleanly and if you got a patch from upstream and stick it into debian/patches/, but it doesn't apply, you have to fix it, of course. cdbs-edit-patch and dpatch-e-p will just try and leave you in the shell with the *.rej files then you simply resolve the *.rej files, and press C-D to update the patch. 
  however, please keep in mind that a dpatch is a patch with a shell script header, not a simple patch (as mentioned above) so to avoid this entirely, you should never just stick an unknown patch into debian/patches. instead, you use *-edit-patch and *then* use patch -p1 < /path/to/your/new/patch. edit the mess there. and let *-edit-patch create the actual patch in the source tree
{{{
06:50	pitti	then let's go to the last patch system
06:50	pitti	== xterm: quilt ==
06:50	pitti	quilt is the other non-dumb standard patch system. Like dpatch, it has a list of patches to apply in patches/series (to use debian/patches, packages need to add a sylink).
06:50	pitti	It is non-trivial to set up and has a lot of advanced commands which make it very flexible, but not very easy to use.
06:50	pitti	nontrivial to set up for Debian source packages, that is
06:50	pitti	(it's not hard either, but more work than simple-patchsys, and even dpatch)
06:50	pitti	I will only show a small example here
06:50	pitti	in the xterm source
06:51	pitti	First, you can use the existing machinery to set up symlinks and directories for quilt:
06:51	pitti	cd /whereever/you/unpacked/the/source/xterm-216
06:51	pitti	debian/rules prepare
06:51	pitti	the 'prepare' target is not standardized; you need to look into debian/rules
06:51	pitti	however, it usually boils down to 'export QUILT_PATCHES=debian/patches' (which should work fine everywhere)
06:51	pitti	since quilt's default patch dir is ./patches
06:51	pitti	so some packages set up a symlink, some export QUILT_PATCHES, etc.
06:52	pitti	Now let's edit the already existing patch 901_xterm_manpage.diff:
06:52	pitti	quilt push 901_xterm_manpage.diff
06:52	pitti	this will apply all patches in the stack up to the given one
06:52	pitti	apply inline right in the source tree, that is
06:52	pitti	i. e. *not* in a temporary directory and not in a subshell
06:52	pitti	quilt keeps track of applied patches in the source tree internally
06:53	pitti	now let's edit a file that is already touched by the original patch
06:53	pitti	  sed -i 's/Copyright/Copyleft/' xterm.man
06:53	pitti	let's commit the change:
06:53	pitti	  quilt refresh 901_xterm_manpage.diff
06:53	pitti	So unlike the other patch systems, quilt works with patched inline sources, but keeps track of modifications.
=== pitti waits a bit for people to catch up and finish the example on their keyboards
06:55	pitti	Finally, let's add a new patch to the top of the stack:
06:55	pitti	  quilt push -a
06:55	pitti	'-a' means 'all patches', thus it applies all further patches after 901_xterm_manpage.diff up to the top
06:55	pitti	  quilt new muhaha.diff
06:55	pitti	register a new patch name (which we want to put on top of the patch stack)
06:55	pitti	  quilt add README
06:55	pitti	you have to do that for all files you modify, so that quilt can keep track of the original version
06:55	pitti	this tells quilt to keep track of the original version of README
06:55	pitti	  sed -i '1 s/^/MUHAHA/' README
06:55	pitti	modify the source
06:56	pitti	  quilt refresh
06:56	pitti	this will finally create debian/patches/muhaha.diff with the changes to README
06:56	pitti	as I already said above, quilt has a patch list, too
06:56	pitti	in debian/patches/series
06:56	pitti	which is much like debian/patches/00list for dpatch
06:56	pitti	and if you push -a, then the patch will land on top of the patch stack, and will automatically be put at the end of series
06:57	pitti	of course you can create the patch in other levels of the patch stack
06:57	pitti	but usually you want the top
06:57	pitti	sometimes, when you pull changes from upstream CVS, it's better to put them at the bottom of the stack
06:57	pitti	i. e. upstream changes shuold generally come *before* distro-specific changes
06:57	pitti	ok?
06:57	jrib	ok
06:58	pitti	=== A glimpse into the future ===
06:58	pitti	As you saw, Debian source packages do not have any requirements wrt. structure, patch systems, etc., other source package systems like SRPM are much stricter wrt that. This of course means more flexibility, but also much more learning overhead.
06:58	pitti	As a member of the security team I can tell tales of the pain of a gazillion different source package layouts... :)
06:58	pitti	Therefore some clever people sat together the other day to propose a new design which would both give us a new and unified source package and patch system that uses bzr (with a quilt-like workflow). This would also integrate packages and patches much better into Launchpad and revision control in general.
06:58	pitti	Please take a look at https://wiki.ubuntu.com/NoMoreSourcePackages if you are interested in this.
06:59	pitti	I don't want to handle this here, since it's really distant future
06:59	pitti	oh, and officially again:
06:59	_ion	That was new to me. Sounds very cool.
06:59	pitti	=== Notes ===
06:59	pitti	The channel topic points to fabbione's logs, so that you can get the full conversation
06:59	pitti	There is also a wiki page https://wiki.ubuntu.com/MOTU/School/PatchingSources which provides most of above information in a more convenient format. However, it might be slightly out of date (it's from dapper times). Feel free to update the page and and add missing bits.
06:59	pitti	we are already at the end of the our
06:59	pitti	thank you for your attention!
06:59	pitti	I hope you could learn a bit
07:01	pitti	if you have any further questions, can you please mail me? martin.pitt at ubuntu.com
}}}
''[jrib]	sorry, one more question about quilt.  After our last command, the patches are still applied so do we have to unapply them before building the source package?''

 debclean will take care of that. and if you build without cleaning, then quilt will know that it already applied the patches. as I said, quilt keeps track of everything, it's pretty hard to break

''[DenisTheMenace]	how can i build the source package?''

 debuild -us -uc -b

''[jorgp]	which patching system do you prefer and why?''

 I use cdbs+simple-patchsys.mk for my packages.  mainly because I like it's simplicity, wrote cdbs-edit-patch which makes it easy to deal with it, and I don't want to use different systems. the concept of factorization and modularization is The Right Thing (tm). if only the docs were better :)