FixingBugs1

Revision 3 as of 2008-09-02 20:01:05

Clear message

Dev Week -- How do I fix an Ubuntu bug? - Daniel Holbach -- Tue, Sep 2

(12:00:37 PM) dholbach: alright... let's get started - my name is Daniel Holbach, I've been working with the MOTU team since hoary and work for Canonical for quite a while, trying to make developing Ubuntu as fun as possible
(12:01:15 PM) dholbach: together we're going to grab some low-hanging fruit and fix a bug together
(12:01:34 PM) dholbach: questions please in #ubuntu-classroom-chat, prefixed with "QUESTION: ..."
(12:02:01 PM) dholbach: for those of you who are participating can you please just mention which Ubuntu release you are on and if you have a fast/slow connection to the net?
(12:02:18 PM) dholbach: in here is fine
(12:02:21 PM) Oli``: 8.10 - 8meg
(12:02:31 PM) mazaalai: hardy - 8meg
(12:02:35 PM) dholbach: just fast/slow is fine :-)
(12:02:41 PM) mruiz: 8.04 fast
(12:02:42 PM) tacone: 8.10 , 10meg
(12:02:42 PM) balachmar: 8.04 fast
(12:02:44 PM) mazaalai: fast :)
(12:02:47 PM) dholbach: I don't want you to start measuring now :-)
(12:02:47 PM) palango: 8.04 fast
(12:02:51 PM) Bijoy: hardy - medium
(12:02:51 PM) riot_le: 8.04 fast
(12:02:52 PM) Oli``: heh
(12:02:56 PM) fluteflute: intrepid, medium
(12:02:59 PM) takdir: 8.10 - slow
(12:03:01 PM) techno_freak: hardy - medium
(12:03:02 PM) Volans: 7.10 + pbuilder-intrepid - 7meg
(12:03:14 PM) hempal: 8.04 Medium (relative)//
(12:03:33 PM) gery: 8.10 - slow
(12:03:40 PM) dholbach: ok.. 8.04, 8.10 and 7.10 should be fine AFAIK - if you run into problems or have questions, please raise it on #ubuntu-classroom-chat
(12:03:53 PM) dholbach: we need to do some preparations first
(12:03:57 PM) sebner: 8.10 16mbit :P
(12:04:13 PM) dholbach: those of you who have a reasonably fast connection will install pbuilder
(12:04:23 PM) dholbach: please do the following:
(12:04:24 PM) dholbach:    sudo apt-get install pbuilder devscripts
(12:04:33 PM) dholbach: then edit    ~/.pbuilderrc
(12:04:40 PM) fabian23_: hardy slow
(12:04:40 PM) dholbach: and put
(12:04:42 PM) dholbach: COMPONENTS="main universe multiverse restricted"
(12:04:43 PM) dholbach: into it
(12:04:48 PM) dholbach: then run
(12:04:53 PM) dholbach:    sudo pbuilder create
(12:05:05 PM) dholbach: we'll let that run in the background as it will take some time
(12:05:19 PM) dholbach: everybody else please just install devscripts
(12:05:44 PM) dholbach: pbuilder is a tool with which we can test-build source packages in a minimal environment
(12:06:12 PM) dholbach: what I just mentioned was the short cut, https://wiki.ubuntu.com/PbuilderHowto has more information
(12:06:38 PM) dholbach: ok... while that's running, let's start looking for those low-hanging fruit :-)
(12:07:27 PM) dholbach: a while ago I started an effort called Harvest - it's basically a webpage that pulls data from various sources and displays that information per package
(12:07:30 PM) dholbach:   http://daniel.holba.ch/harvest
(12:07:44 PM) dholbach: <tacone> QUESTION: which distro pbuilder should have created ?
(12:07:54 PM) dholbach: tacone: the ones that were mentioned before should all be fine
(12:08:26 PM) dholbach: if you click on the "Sourcepackage list" link it will present you with an awfully long list of "opportunities"
(12:08:44 PM) dholbach: let's fast-forward to http://daniel.holba.ch/harvest/handler.py?pkg=djvulibre
(12:08:51 PM) dholbach: <mazaalai> QUESTION: how I can use local mirror server in pbuilder instead of archive.ubuntu.com
(12:09:05 PM) dholbach: mazaalai: please check out https://wiki.ubuntu.com/PbuilderHowto - it should have that information
(12:09:48 PM) dholbach:  MIRRORSITE="http://us.archive.ubuntu.com/ubuntu" I think
(12:10:03 PM) dholbach: ok, everybody at   http://daniel.holba.ch/harvest/handler.py?pkg=djvulibre   right now?
(12:10:09 PM) dholbach: it shows three opportunities
(12:10:12 PM) dholbach: 2 fedora patches
(12:10:22 PM) dholbach: and one from the "opportunities" list
(12:10:25 PM) dholbach: errr
(12:10:29 PM) dholbach: and one from the "patches" list
(12:10:43 PM) dholbach: please click on the  255695  link
(12:10:55 PM) dholbach: patches means: this is a bug with a patch attached
(12:11:13 PM) dholbach: as you can see, Harvest makes it easy to gather all that kind of information about packages that you're interested in
(12:11:41 PM) dholbach: amont them are: patches in Launchpad, upgrade requests, fedora patches, if the package does not build from source right now, etc etc
(12:12:23 PM) dholbach: sometimes it takes a bit to find something suitable to work on (because you don't know the package well enough or the bug is too complicated and so on), but Harvest is a good start to find those low hanging fruit
(12:12:51 PM) dholbach: ok... the bug report says something's wrong with the manpage of c44 (which is included in the package)
(12:13:04 PM) dholbach: let's see if that's true, please run
(12:13:07 PM) dholbach:    dget http://daniel.holba.ch/motu/djvulibre_3.5.20-7ubuntu1.dsc
(12:13:53 PM) dholbach: for those of you who attended  Packaging 101  yesterday, you already notice which files were downloaded and what they are their for
(12:14:04 PM) dholbach: <balachmar> QUESTION: do I want to do dget in a seperate folder, or does it put it into the pbuilder stuff?
(12:14:15 PM) dholbach: balachmar: as you like it, you can run it from a special directory if you like, no problemo
(12:14:39 PM) DarkSyranus: ls
(12:14:50 PM) dholbach: it downloaded a .orig.tar.gz a .dsc and a .diff.gz file
(12:15:48 PM) dholbach: short version: .orig.tar.gz is the unmodified tarball that the software authors released on their homepage, .diff.gz the compressed set of changes we need to apply to make it build "our way" and become a nice .deb package in the end, the .dsc file contains metadata like md5sum, etc
(12:16:07 PM) dholbach: alright
(12:16:33 PM) dholbach: please run    dpkg-source -x djvulibre_3.5.20-7ubuntu1.dsc
(12:16:51 PM) dholbach: this will extract the tarball, then apply the compressed patch
(12:16:57 PM) dholbach: <mitesh> QUESTION: it says "dscverify: can't find any Debian keyrings "?
(12:17:04 PM) dholbach: mitesh: it's a warning, you can safely ignore it
(12:18:17 PM) dholbach: <fluteflute> QUESTION: I get dpkg-source: error: File ./djvulibre_3.5.20.orig.tar.gz has size 1359872 instead of expected 2426487 ?
(12:18:25 PM) dholbach: fluteflute: sorry for that, I'm just fixing the mistake
(12:19:46 PM) dholbach: sorry for that everybody, please run the commands again
(12:19:57 PM) dholbach: that is:
(12:19:58 PM) dholbach:   dget http://daniel.holba.ch/motu/djvulibre_3.5.20-7ubuntu1.dsc
(12:20:06 PM) dholbach:   dpkg-source -x djvulibre_3.5.20-7ubuntu1.dsc
(12:20:46 PM) dholbach: thanks for bearing with me :)
(12:20:50 PM) dholbach: OK
(12:20:50 PM) mruiz: :)
(12:21:00 PM) mitesh: :)
(12:21:10 PM) dholbach: let's try to find out if the problem that Mr Ralph Corderoy describes is still valid
(12:21:30 PM) dholbach: cd djvulibre-3.5.20/tools
(12:21:49 PM) dholbach: nroff -man c44.1 | less
(12:22:03 PM) dholbach: this will display the manpage
(12:22:32 PM) dholbach: and since he alread mentions that it's around the "PPM" bit in the text, we can run:
(12:22:35 PM) KennethVenken: ls
(12:22:40 PM) dholbach: nroff -man c44.1 | grep -a1 -b1 PPM
(12:22:54 PM) dholbach: you should see the ".SM" bit he is referring to in the bug report
(12:23:04 PM) dholbach: everybody can see that?
(12:23:34 PM) dholbach: let me rephrase: can you guys see that too? :)
(12:23:50 PM) riot_le: yes
(12:23:52 PM) techno_freak: yes
(12:23:54 PM) mitesh: yes
(12:23:56 PM) swingnjazz: yes
(12:23:56 PM) Oli``: yeehaw
(12:24:03 PM) balachmar: yes
(12:24:04 PM) chombium: yes
(12:24:07 PM) dholbach: great
(12:24:13 PM) tacone: I get  troff: fatal error: can't open `c44.1': No such file or directory
(12:24:14 PM) dholbach: just wanted to make sure I hadn't lost you on the way :)
(12:24:26 PM) dholbach: tacone: are you in djvulibre-3.5-20/tools ?
(12:24:34 PM) dholbach: djvulibre-3.5.20/tools
(12:24:40 PM) tacone: no ok, my fault, sorry
(12:24:49 PM) dholbach: alright, let's go on
(12:25:03 PM) dholbach: let's all download the patch that Ralph Corderoy is suggesting
(12:25:33 PM) dholbach: I'll assume you put it into ~ for now :)
(12:25:47 PM) dholbach: please run
(12:25:58 PM) dholbach:   patch -p0 < ~/c44.1.patch
(12:26:04 PM) dholbach: in my case it safely applied
(12:26:09 PM) dholbach: anybody got any problems?
(12:26:25 PM) mazaalai: it must be in djvulibre-3.5.20/tools, right?
(12:26:50 PM) dholbach: mazaalai: no, it doesn't matter where you put the patch, it's just important that you are in the directory, when you run the patch command
(12:27:13 PM) techno_freak: dholbach, successful == .PM shouldn't be there?
(12:27:20 PM) dholbach: techno_freak: I'm coming to that :)
(12:27:29 PM) dholbach: <mitesh> QUESTION : it says bash: /home/mitesh/c44.1.patch: No such file or directory
(12:27:41 PM) dholbach: mitesh: did you download the patch from the bug report to ~?
(12:27:41 PM) chombium: <chombium>QUESTION: where can i download the patch from?
(12:27:55 PM) dholbach: https://bugs.launchpad.net/ubuntu/+source/djvulibre/+bug/255695
(12:27:57 PM) ubot5: Launchpad bug 255695 in djvulibre "c44(1) man page outputs splurious `.SM' macro invocation" [Undecided,New]
(12:28:10 PM) dholbach: the patch operation should take a second at best
(12:28:19 PM) techno_freak:  wget -c http://launchpadlibrarian.net/16614829/c44.1.patch
(12:28:35 PM) dholbach: ok... once you've applied the patch please run
(12:28:44 PM) dholbach:   nroff -man c44.1 | grep -a1 -b1 PPM
(12:28:45 PM) dholbach: again
(12:28:55 PM) dholbach: can you still the ".SM" bit there?
(12:29:06 PM) riot_le: no
(12:29:13 PM) mazaalai: nope
(12:29:14 PM) okar_: no :D
(12:29:17 PM) vishr: done...no
(12:29:19 PM) chombium: nope
(12:29:21 PM) swingnjazz: ok, it's gone
(12:29:21 PM) takdir: no
(12:29:29 PM) tacone: gone
(12:29:32 PM) dholbach: excellent, seems that Ralph Corderoy has done good work :)
(12:29:36 PM) dholbach: ok... now
(12:30:01 PM) dholbach: as we want to fix the bug in Ubuntu we need to add a changelog entry to explain what we changed and why
(12:30:12 PM) dholbach: this all happens in debian/changelog
(12:30:15 PM) dholbach: so please run:
(12:30:16 PM) dholbach:   cd ..
(12:30:18 PM) dholbach:   dch -i
(12:30:38 PM) dholbach: this will fire up an editor with a template changelog entry
(12:31:04 PM) dholbach: I won't go through all the specifics in it, read the Packaging 101 log from yesterday or check out the videos on http://youtube.com/ubuntudevelopers
(12:31:18 PM) dholbach: there's also https://wiki.ubuntu.com/PackagingGuide to find out more about it
(12:32:00 PM) dholbach: just make sure that the top entry says "intrepid" in the first line and that your mail address and name is correct
(12:32:25 PM) dholbach: <Oli``> QUESTION: how do you change the dhc template so it uses the correct email?
(12:32:30 PM) fabian23_: and for those running hardy and gutsy?
(12:32:52 PM) Volans: fabian23_:  dch -i -D intrepid
(12:33:10 PM) dholbach: Oli`` is right, I should have explained that before, for now just edit it manually, I'll tell you later how to fix it for the next times you're going to use the tool
(12:33:24 PM) dholbach: fabian23_ asks an important question: why not hardy, why intrepid?
(12:33:38 PM) dholbach: because we can just make changes in the current development release which is intrepid
(12:33:51 PM) dholbach: all the other releases have been closed
(12:34:16 PM) dholbach: there's the hardy-updates process which works differently, for now the link https://wiki.ubuntu.com/StableReleaseUpdates should be enough
(12:34:25 PM) dholbach: ok, let's document our changes
(12:34:46 PM) dholbach: that's the crucial step, because our fellow developers should not have to guess where our ideas came from
(12:34:55 PM) dholbach: I'll put in something like this:
(12:35:26 PM) dholbach:   * tools/c44.1: applied patch from Ralph Corderoy to fix the manpage markup. (LP: #255695)
(12:35:35 PM) dholbach: three things are worth noting:
(12:35:46 PM) dholbach:  1) explicitly named the files that were changed in the upload
(12:36:00 PM) dholbach:  2) I gave credit to the person who provided the fix
(12:36:13 PM) dholbach:  3) I mentioned which bug report in Launchpad this is all about
(12:36:39 PM) dholbach: in addition to that (LP: #<bugnumber>) will automatically close the bug once it got uploaded to the build daemons
(12:36:48 PM) dholbach: <tacone> QUESTION: shouldn't we use a patch system ?
(12:36:50 PM) dholbach: tacone: good question
(12:37:17 PM) dholbach: there's going to be another session on Sep 3rd at 20:00 UTC about patch systems
(12:37:33 PM) dholbach: this package does not use a patch system itself and directly applies changes to the source, so we'll do the same
(12:38:21 PM) dholbach: once we're done with that, please save the file
(12:38:26 PM) dholbach: and run     debuild -S
(12:39:07 PM) dholbach: this will rebuild the source package (not build the package itself) and refresh the .diff.gz
(12:39:12 PM) dholbach: if you run
(12:39:13 PM) dholbach:   ls ..
(12:39:24 PM) dholbach: you will notice that there are two .diff.gz files now
(12:39:34 PM) dholbach: the one we downloaded and the second one we created ourselves
(12:40:00 PM) dholbach: since the question about patch system comes up again in #ubuntu-classroom-chat:
(12:40:31 PM) dholbach: if the Debian package we're attempting to fix does not add a patch system itself and patches the source inline (as opposed to adding patch files in debian/patches) we will do the same
(12:40:38 PM) dholbach: <tacone> dholbach:  running debsign failed
(12:40:45 PM) dholbach: <Coper> QUESTION: why dosen't debuild -S find my private gpg key?
(12:40:56 PM) dholbach: that's to be expected, I'll enlighten you about it in a bit - it's safe to ignore right now
(12:40:56 PM) chombium: <chombium>QUESTION: /bin/bash: dh_testdir: command not found where can i find it? which package?
(12:41:09 PM) dholbach: chombium: sorry.....    sudo apt-get install debhelper
(12:41:17 PM) chombium: tx
(12:41:23 PM) dholbach: ok
(12:41:32 PM) dholbach: now please run:
(12:42:12 PM) dholbach:   cd ..; debdiff djvulibre_3.5.20-7ubuntu{1,2}.dsc > djvulibre.debdiff
(12:42:38 PM) dholbach: debdiff is a great utility that will compare the two revisions of debian source packages and print out the diff
(12:43:35 PM) dholbach: can you all please go to http://paste.ubuntu.com and paste the contents of your djvulibre.debdiff into it and give the link to your sourcepackage here?
(12:44:07 PM) techno_freak: fluteflute, ^^
(12:44:55 PM) dholbach: who managed to produce a  djvulibre.debdiff ?
(12:45:02 PM) Coper: dholbach: http://paste.ubuntu.com/42746/
(12:45:06 PM) dholbach: can you please post the contents of that file on http://paste.ubuntu.com and give the link here?
(12:45:06 PM) tacone: dholbach: http://paste.ubuntu.com/42747/
(12:45:08 PM) riot_le: http://paste.ubuntu.com/42748/
(12:45:34 PM) dholbach: Coper: looks good!
(12:45:44 PM) vishr: http://paste.ubuntu.com/42752/
(12:45:52 PM) dholbach: tacone: looks good
(12:45:52 PM) palango: http://paste.ubuntu.com/42749/
(12:46:09 PM) balachmar: http://paste.ubuntu.com/42754/
(12:46:09 PM) KennethVenken: http://paste.ubuntu.com/42750/
(12:46:15 PM) dholbach: riot_le: looks good too
(12:46:16 PM) swingnjazz: http://paste.ubuntu.com/42753/
(12:46:28 PM) Oli``: http://paste.ubuntu.com/42755/
(12:46:36 PM) Bijoy: http://paste.ubuntu.com/42756/
(12:46:45 PM) techno_freak: http://paste.ubuntu.com/42757/
(12:46:47 PM) dholbach: vishr: looks good as well, I'd just indent the 2nd line of the changelog entry a bit
(12:47:05 PM) dholbach: palango: the same goes for yours
(12:47:18 PM) palango: ok
(12:47:31 PM) dholbach: balachmar: yours is OK, I'd just wrap the line in the changelog entry
(12:47:45 PM) dholbach: KennethVenken: same as balachmar
(12:47:50 PM) takdir: http://paste.ubuntu.com/42758/
(12:47:55 PM) chombium: http://paste.ubuntu.com/42759/
(12:48:07 PM) dholbach: swingnjazz: please the contents of the file, not the command line output
(12:48:17 PM) dholbach: Oli``: same as balachmar
(12:48:25 PM) balachmar: @dholbach the famous 60 characters or something?
(12:48:30 PM) dholbach: Bijoy: was that the whole file?
(12:48:32 PM) swingnjazz: uuh, sorry for that
(12:48:46 PM) dholbach: balachmar: yeah, editors like vi make it easier for you
(12:48:57 PM) dholbach: techno_freak: same as balachmar's :)
(12:49:13 PM) dholbach: takdir: your changelog entry is a bit short
(12:49:24 PM) dholbach: chombium: your changelog entry is empty
(12:49:25 PM) techno_freak: dholbach, ok
(12:49:31 PM) dholbach: other than that:  W E L L   D O N E
(12:49:33 PM) dholbach: I mean...
(12:49:43 PM) dholbach:               _ _       _
(12:49:43 PM) dholbach: __      _____| | |   __| | ___  _ __   ___
(12:49:43 PM) dholbach: \ \ /\ / / _ \ | |  / _` |/ _ \| '_ \ / _ \
(12:49:43 PM) dholbach:  \ V  V /  __/ | | | (_| | (_) | | | |  __/
(12:49:43 PM) dholbach:   \_/\_/ \___|_|_|  \__,_|\___/|_| |_|\___|
(12:49:44 PM) dholbach:                                            
(12:49:46 PM) dholbach: everybody!
(12:49:58 PM) dholbach: <Volans> QUESTION: I see that in the debdiff there are entries also for autogenerated files, we should leave them or use filterdiff?
(12:50:01 PM) dholbach: Volans: good question
(12:50:14 PM) Bijoy: sorry, my bad, I'll paste again
(12:50:27 PM) dholbach: config.guess and config.sub are build helpers that were automatically updated and can be filtered out - it makes reviewing a lot easier
(12:50:36 PM) dholbach: <Oli``> QUESTION: Is there a reason why there's a ~200 line difference in diff size between some of ours?
(12:50:42 PM) dholbach: Oli``: that's due to what I just said above
(12:50:51 PM) dholbach: <balachmar> REMARK: But the command dch -i started nano not vi
(12:50:53 PM) dholbach: balachmar: you're right
(12:51:06 PM) dholbach: so let's fix all the GPG, Name, Email address, Editor issues you all saw before
(12:51:34 PM) dholbach: if you all use bash as your shell (the default), then please edit ~/.bashrc
(12:51:36 PM) swingnjazz: http://paste.ubuntu.com/42762/
(12:51:43 PM) Bijoy: http://paste.ubuntu.com/42761/
(12:51:51 PM) dholbach: and add something like this at the end of it:
(12:51:53 PM) dholbach: export DEBFULLNAME='Daniel Holbach'
(12:51:53 PM) dholbach: export DEBEMAIL='daniel.holbach@ubuntu.com'
(12:51:53 PM) dholbach: export EDITOR=vim
(12:51:59 PM) dholbach: then save the file and run:
(12:52:03 PM) dholbach:   source ~/.bashrc
(12:52:19 PM) dholbach: swingnjazz: better, I'd just wrap the line of the changelog entry
(12:52:43 PM) dholbach: Bijoy: I'd indent the second line of the changelog entry a bit
(12:52:48 PM) dholbach: other than that: GREAT
(12:52:57 PM) dholbach: <balachmar> QUESTION: So what to do now we have patched it?
(12:53:03 PM) dholbach: that's the good question :-)
(12:53:16 PM) dholbach: let's build the package and see if after our changes it still builds
(12:53:23 PM) dholbach: if your pbuilder command has finished, please run
(12:53:36 PM) dholbach:   sudo pbuilder build djvulibre_3.5.20-7ubuntu2.dsc
(12:53:50 PM) dholbach: this is going to take a while
(12:54:03 PM) dholbach: in the meantime, I'm going to explain once you've found that:
(12:54:08 PM) dholbach:  1) the package still builds
(12:54:22 PM) dholbach:  2) once you installed the new package, your system still works
(12:54:26 PM) dholbach:  3) all is good
(12:54:39 PM) dholbach: (testing is crucial, but we don't do it in the 6 remaining minutes)
(12:55:16 PM) dholbach: what happens once all tests passed and you're happy with everything?
(12:55:24 PM) dholbach: you'll follow the sponsoringship process
(12:56:01 PM) dholbach: Sponsoring means: somebody who has upload privileges already will review your debdiff (you'll attach it to the bug in question) and if they're happy with what you've done, sign the source package with their GPG key and upload it
(12:56:14 PM) dholbach: https://wiki.ubuntu.com/SponsorshipProcess has more information about that
(12:56:32 PM) dholbach: (all the links I mentioned before are on https://wiki.ubuntu.com/MOTU/GettingStarted <- that's the one you should bookmark)
(12:56:40 PM) dholbach: <balachmar> QUESTION: Is there a way to mark a bug as being worked upon?
(12:56:44 PM) dholbach: very important question
(12:56:46 PM) dholbach: yes, there is
(12:57:19 PM) dholbach: you will click on the little arrow on the yellow bar in the middle of the bug report
(12:57:35 PM) dholbach: and set yourself as the assignee and set it to in progress
(12:58:22 PM) dholbach: as you can see: fixing Ubuntu is not always rocket science, but more a matter of detective skills and careful testing and asking questions
(12:58:37 PM) dholbach: the door to #ubuntu-motu is always open and you can ask questions there
(12:59:11 PM) dholbach: please read the sponsorship page carefully when asking for sponsorship
(12:59:19 PM) dholbach: any more, maybe general questions? :)
(12:59:23 PM) dholbach: we have a minute left :)
(12:59:49 PM) dholbach: OK, we'll have a nother session on thursday
(12:59:54 PM) dholbach: fixing bugs again :-)
(01:00:04 PM) dholbach: thanks everybody, you really ROCK
(01:00:06 PM) riot_le: do you publish this HowTo in the Wiki too?
(01:00:08 PM) sebner: but important ones since FF is here :P
(01:00:28 PM) techno_freak: thanks a lot dholbach :)
(01:00:33 PM) riot_le: +1
(01:00:36 PM) balachmar: thanks!
(01:00:39 PM) mazaalai: thanks dholbach
(01:00:39 PM) tacone: \o/
(01:00:42 PM) vishr: thanks a lot dholbach
(01:00:44 PM) dholbach: I'll repeat what I said yesterday: I'd like to see your names connected to Ubuntu Development soon, so make me proud! :-)
(01:00:46 PM) swingnjazz: thanks to you
(01:00:48 PM) balachmar: I am going to find some fruit straight away!
(01:01:03 PM) Bijoy: Thank you Daniel!
(01:01:22 PM) dholbach: next up is the unstoppable David Futcher (bobbo) who is going to introduce you to bzr
(01:01:28 PM) dholbach: thanks everybody!