Dev Week -- Daily Builds and You -- JorgeCastro -- Wed, Jul 14th, 2010
(02:00:49 PM) jcastro: alright!
(02:00:56 PM) dholbach: hello everybody :)
(02:01:04 PM) jcastro: welcome to our class
(02:01:14 PM) jcastro: this session will be about daily builds, I'm Jorge Castro ...
(02:01:23 PM) dholbach: and I'm Daniel Holbach
(02:01:41 PM) jcastro: and we're working with the launchpad team to make daily builds a reality
(02:01:47 PM) jcastro: before I start
(02:01:57 PM) jcastro: let me give you some information that you need to follow along in this class
(02:02:03 PM) jcastro: https://help.launchpad.net/Packaging/SourceBuilds
(02:02:13 PM) jcastro: and all the pages in that section of the wiki will help you out
(02:02:23 PM) jcastro: so first off, what the heck is a daily build?
(02:02:24 PM) dholbach: Yes, https://help.launchpad.net/Packaging/SourceBuilds/GettingStarted is the page you should probably bookmark :)
(02:02:32 PM) jcastro: simply put
(02:02:49 PM) jcastro: a daily build is a PPA that contains software from a project that is updated every day
(02:03:11 PM) jcastro: so for example, if we have daily builds of gedit, then the daily build will have the code from that day in the PPA
(02:03:18 PM) jcastro: so why would we do this?
(02:03:29 PM) jcastro: First of all, the main reason is testing
(02:03:37 PM) jcastro: if you're writing a program, you want to be able to test it
(02:04:00 PM) jcastro: however because ubuntu releases are stable (meaning we don't update to new versions generally)
(02:04:06 PM) jcastro: then this can be hard to get people to test.
(02:04:29 PM) jcastro: ideally we want any upstream project to be able to fix a bug and enable users to check to see if it's fixed
(02:04:38 PM) jcastro: and we plan to do this by offering the latest builds of things
(02:04:54 PM) jcastro: it can be annoying when a developer tells you "try it with this patch, let me know if it works!"
(02:05:07 PM) jcastro: with daily builds we can now easily test that software
(02:05:18 PM) jcastro: and enable the upstream developer to get those fixes out to their users
(02:05:22 PM) jcastro: which is what it's all about
(02:05:39 PM) jcastro: however, there can be badness too, it's not all cupcakes and unicorns
(02:05:54 PM) jcastro: if you commit broken code, it shows up in the daily build
(02:06:06 PM) jcastro: since dailies are automated, if you break your code, this can break the software
(02:06:23 PM) jcastro: some well run projects use things like distributed version control and unit testing to make sure that trunk always builds
(02:06:28 PM) jcastro: this is a Good Thing(tm)
(02:06:54 PM) jcastro: also, if your project is starting out and you're not ready for feedback then dailies can be a waste for you while you make major changes
(02:07:12 PM) jcastro: or if you rewrite entire chunks and don't have a good way for users to move back and forth between versions
(02:07:31 PM) jcastro: and lastly, the world is full of people who love crack, and might use daily builds for day-to-day usage instead of testing
(02:07:50 PM) jcastro: so you might get some bad feedback from people complaining that the software is broken half the time
(02:07:53 PM) jcastro: but whatever. :D
(02:08:04 PM) jcastro: ok so now that's daily builds
(02:08:10 PM) jcastro: now Daniel is going to show you how they work.
(02:08:12 PM) dholbach: and with a PPA it's incredibly easy to get users to test software, if they're interested in that particular package, it's just a "sudo add-apt-repository …" away (or well they use the GUI option :-)), so for an upstream or somebody who "adopted an upstream" it couldn't be easier to get willing testers :)
(02:09:18 PM) dholbach: I wanted to add a note on "warning users", etc. because I think it's very important that you as an upstream or you as somebody who wants to test a particular piece of software tell people very carefully what's going to happen :)
(02:10:00 PM) dholbach: in https://help.launchpad.net/Packaging/SourceBuilds/KnowledgeBase we have a bunch of good information on how to tell people what's happening, which cases it makes sense to have daily builds etc
(02:10:14 PM) dholbach: https://help.launchpad.net/Packaging/SourceBuilds/GoingBack for example explains in a visual way how to "go back to an old version"
(02:10:25 PM) dholbach: are there any questions up until now?
(02:10:59 PM) dholbach: alright, seems what Jorge and I said made at least a bit of sense :-)
(02:11:23 PM) dholbach: first of all: this feature is BETA and we have already found a few bugs that still need to be ironed out
(02:11:29 PM) dholbach: https://help.launchpad.net/Packaging/SourceBuilds/KnownLimitations is what's on our list right now
(02:11:40 PM) dholbach: ok, so how does this work
(02:11:51 PM) dholbach: first of all: all the code needs to be in launchpad
(02:12:12 PM) dholbach: so if you're interested in some project that is hosted somewhere in a cvs repository somewhere, you need to import it into launchpad
(02:12:16 PM) dholbach: luckily, that's very easy
(02:12:28 PM) dholbach: we have links in the knowledge base about how to do exactly that
(02:12:30 PM) jcastro: (... we also support svn and git imports)
(02:12:44 PM) dholbach: yes, that's all explained in there :)
(02:12:50 PM) dholbach: ah, we have the first few questions
(02:12:58 PM) dholbach: <joaopinto> QUESTION: Are daily builds targeted for every supported release ?
(02:13:11 PM) dholbach: joaopinto: you can target them to releases as you like it
(02:13:16 PM) dholbach: <simar_mohaar> QUESTION: Whats a PPA?
(02:13:31 PM) dholbach: simar_mohaar: it's a Personal Package Archive
(02:13:46 PM) dholbach: in addition to the ubuntu archive people or teams can set up their own archives in Launchpad
(02:13:55 PM) dholbach: https://help.launchpad.net/PPA has more detail about this
(02:14:00 PM) dholbach: it's the best thing since sliced bread
(02:14:12 PM) dholbach: ok, now that we have the code in Launchpad, what's next
(02:14:21 PM) dholbach: there's the "recipe"
(02:14:46 PM) dholbach: this gets a little bit more complicated, so hold tight and relax - we'll get to the actual builds in a bit :)
(02:15:24 PM) dholbach: a recipe is instructions how launchpad is going to assemble the source package on its own
(02:15:37 PM) dholbach: (from which source branch, how does the version numbering work, etc.)
(02:15:54 PM) dholbach: it's crucial that you write up that recipe (we have a few nice examples) and TEST it locally
(02:16:23 PM) dholbach: and when I mean test
(02:16:24 PM) dholbach: I mean
(02:16:25 PM) dholbach: _____ _____ ____ _____ _
(02:16:25 PM) dholbach: |_ _| ____/ ___|_ _| |
(02:16:25 PM) dholbach: | | | _| \___ \ | | | |
(02:16:25 PM) dholbach: | | | |___ ___) || | |_|
(02:16:25 PM) dholbach: |_| |_____|____/ |_| (_)
(02:16:27 PM) dholbach:
(02:16:34 PM) dholbach: luckily we have tools for that and we'll cover that in a bit
(02:16:50 PM) jcastro: ok so let's start with a simple recipe
(02:16:56 PM) jcastro: # bzr-builder format 0.2 deb-version {debupstream}+{revno}
(02:16:56 PM) jcastro: lp:gtg
(02:16:57 PM) jcastro: nest packaging lp:~gtg/gtg/debian debian
(02:17:00 PM) jcastro: 3 lines!
(02:17:04 PM) jcastro: let's go over each
(02:17:11 PM) jcastro: the first one defines the recipe
(02:17:22 PM) jcastro: the 0.2 is the version you want the package to be
(02:17:29 PM) jcastro: I usually derive this from the upstream version
(02:17:45 PM) jcastro: (remember that you need the version numbers to be close to what upstream is
(02:18:16 PM) jcastro: actually, wait
(02:18:19 PM) jcastro: I totally messed that up
(02:18:26 PM) jcastro: let's start with line 1 again
(02:18:38 PM) jcastro: "bzt-builder format" is needed to start the recipe
(02:18:50 PM) jcastro: 0.2 is the version of the recipe formate
(02:18:58 PM) jcastro: so for those you can just leave those the same
(02:19:36 PM) dholbach: deb-version describes what the resulting package in the PPA will have as version number
(02:19:53 PM) dholbach: you need to be a bit careful there
(02:20:15 PM) dholbach: {debupstream} will be replaced with the version number in the packaging (debian/changelog - more on that in a sec)
(02:20:35 PM) dholbach: {revno} is the revision number of the upstream branch
(02:20:47 PM) jcastro: the second line is easy
(02:20:53 PM) jcastro: it's where in launchpad the upstream source is
(02:21:08 PM) jcastro: so in this example, since gtg is hosted in launchpad, we can use lp:gtg
(02:21:14 PM) jcastro: for imports it might be more comples
(02:21:18 PM) jcastro: complex I mean
(02:21:32 PM) jcastro: then the last line is where the packaging is
(02:22:13 PM) jcastro: so we're basically telling launchpad "grab the code from lp:gtg, and nest the packaging from lp:~gtg/gtg/debian which is a debian directory, and spit it out."
(02:22:31 PM) dholbach: so in this case we have the following case
(02:22:40 PM) dholbach: lp:gtg contains no packaging at all
(02:22:44 PM) dholbach: it's just the upstream source
(02:23:21 PM) dholbach: usually if you get the source of a source package, you can see a debian/ directory in there which contains all the packaging information
(02:23:27 PM) dholbach: so on its own lp:gtg would not build
(02:24:02 PM) dholbach: so what we do here is: check out a branch that contains the current packaging and stick it in to a debian/ directory in what we checked out as trunk
(02:24:38 PM) dholbach: the second branch does ONLY contain packaging, nothing else
(02:24:48 PM) dholbach: so "nest" is the keyword that means "ok, check out this second branch and just put it where I tell you"
(02:25:19 PM) dholbach: ok, that was case one and what we think is a very common one
(02:25:32 PM) dholbach: for case two, I'd like to highlight fabrice_sp's question :)
(02:25:39 PM) dholbach: <fabrice_sp> QUESTION: what if upstream ships a debian directory?
(02:26:12 PM) dholbach: that case is simple: you just drop the third line, you have a two line recipe then :-D
(02:26:43 PM) dholbach: and now we have case three which is a bit more complicated - I hope you're a little bit familiar with distributed revision control, so I don't look stupid
(02:27:12 PM) dholbach: let's recap: case 1) pristine trunk, does not have packaging, so "nest" packaging branch into debian/ dir
(02:27:23 PM) dholbach: case 2) upstream has packaging already, so you get a simple recipe
(02:27:25 PM) dholbach: case 3)
(02:27:41 PM) dholbach: you have two branches you're going to merge
(02:27:56 PM) dholbach: so for example you have the upstream trunk that does not have any packaging included
(02:28:49 PM) dholbach: and the other branch was branched off of upstream and you added packaging in there, maybe you just wanted to ship a particular release
(02:29:21 PM) dholbach: as you can see the third one is a bit more complicated, but as I said we have tools to test and see if it works :-)
(02:29:26 PM) jcastro: hey wait a minute!
(02:29:31 PM) jcastro: so let's say I have my upstream
(02:29:33 PM) jcastro: and the packaging
(02:29:39 PM) jcastro: but someone out there made bug fixes in a branch
(02:29:48 PM) jcastro: I want to be able to try those fixes in a build
(02:29:56 PM) jcastro: so can I merge in branches with fixes from launchpad?
(02:30:00 PM) jcastro: like say ....
(02:30:05 PM) jcastro: merge fix-build lp:~bzr/bzr/fix-build
(02:30:16 PM) dholbach: exactement - that's exactly what you can do now :)
(02:30:36 PM) jcastro: so we can combine stuff!
(02:30:43 PM) dholbach: yeah :-)
(02:30:44 PM) jcastro: so let's check out a more convoluted example
(02:30:51 PM) jcastro: # bzr-builder format 0.2 deb-version 1.0+{time}
(02:30:51 PM) jcastro: lp:bzr
(02:30:51 PM) jcastro: merge fix-build lp:~bzr/bzr/fix-build
(02:30:51 PM) jcastro: nest pyfoo lp:pyfoo foo
(02:30:51 PM) jcastro: merge branding lp:~bob/pyfoo/ubuntu-branding
(02:30:53 PM) jcastro: merge packaging lp:~bzr/bzr/packaging
(02:31:16 PM) dholbach: wow :)
(02:31:26 PM) jcastro: I can even specify revisions!
(02:31:30 PM) jcastro: merge packaging lp:~bzr/bzr/packaging revno:2355
(02:31:39 PM) jcastro: whoa, now I see why this is so powerful! :)
(02:31:46 PM) dholbach: yeah, that takes a bit to understand, but if you do it step by step, it'll all be nice and easy
(02:31:51 PM) dholbach: as an added bonus you get an email if the fixed branch does not apply any more in trunk :)
(02:32:13 PM) dholbach: <joaopinto> QUESTION: Is there some middle ground official project, technically similar to daily builds but providing stable releases instead ?
(02:32:43 PM) dholbach: joaopinto: good question: sometimes upstream will have a stable branch, so you could have multiple PPAs: one for beta, one for stable, one for crack of the day, etc.
(02:33:02 PM) dholbach: in the other cases, you could supply revision numbers, etc.
(02:33:19 PM) dholbach: this technology is still quite new, but we'll figure out over time how to make the best use of it
(02:33:25 PM) jcastro: so if we have the packaging, the upstream, and PPAs ... then really it's up to me if I want to publish it daily
(02:33:26 PM) dholbach: right now we still have to do it carefully
(02:33:36 PM) jcastro: or just follow stable branches
(02:33:39 PM) dholbach: jcastro: exactly :)
(02:33:51 PM) jcastro: ok so before we go off and flood lp
(02:33:58 PM) jcastro: can you tell me why it's so important to test locally?
(02:34:01 PM) jcastro: and then show me how?
(02:34:15 PM) dholbach: sure, will do :)
(02:35:20 PM) dholbach: there's still a couple of bugs the Soyuz team (which does the building and PPA part of things), the bazaar team (who deals with all the branch goodness) and the launchpad-code team (the folks who work on merge proposals, code import, etc.) and lots of others are working on
(02:35:38 PM) dholbach: plus, there's currently a bunch of builds in PPAs piled up
(02:35:45 PM) jcastro: (I hear they're sprinting right now to fix these bugs!)
(02:36:09 PM) dholbach: so if you know the build is not really important or you don't know it works, it's unfair to others if this hammers launchpad
(02:36:26 PM) jcastro: you mean I can't just set up 50 of them and then just fire and forget?
(02:37:08 PM) dholbach: I don't know if there's any limit hardcoded in LP, but we should all bear in mind that it's BETA, that it's a service that other folks use and that we should all take it nice and easy :)
(02:37:21 PM) dholbach: matttbe37 asked a question about this too
(02:37:22 PM) dholbach: <matttbe37> QUESTION: Hello! Do you know when the version of 'bzr-builder' will be updated in launchpad? => because it seems there is a bug with the current version (Bug #604837) and also because I need the new {date} variable :) ?
(02:37:44 PM) dholbach: matttbe37: honestly I don't know right now, but I'll make sure to get some feedback on it and make sure that it's addressed somehow
(02:37:56 PM) dholbach: alright
(02:38:01 PM) dholbach: let's get to testing it
(02:38:34 PM) dholbach: this is not a demo or packaging session, so we'll just give you the quick instructions here and you can go and try this out later on
(02:38:42 PM) dholbach: basically you write your recipe based on the instructions
(02:38:48 PM) dholbach: then you save it locally
(02:38:50 PM) dholbach: then install bzr-builder
(02:38:54 PM) jcastro: https://launchpad.net/builders has build stats (and yes there are more resources coming)
(02:39:19 PM) dholbach: and then you just run "bzr dailydeb package.recipe working-dir"
(02:39:33 PM) dholbach: which will try out just what launchpad is going to do for you
(02:39:46 PM) dholbach: (it'll get the branches you specified and assemble the source package for you)
(02:40:09 PM) jcastro: (this way you can also build the packages locally and make sure they work before asking launchpad to do it for you)
(02:40:11 PM) dholbach: there's one particular big guy in the launchpad team (bigjools) who'll be unhappy if you don't try this out first
(02:40:16 PM) dholbach: so don't say I didn't warn you :)
(02:40:40 PM) dholbach: then you can use pbuilder to test build the package locally
(02:41:00 PM) dholbach: and that's a good exercise as well :)
(02:41:04 PM) dholbach: and it's easy to do
(02:41:09 PM) dholbach: sudo apt-get install pbuilder
(02:41:26 PM) dholbach: add COMPONENTS="main universe multiverse restricted" to ~/.pbuilderrc
(02:41:37 PM) dholbach: sudo pbuilder create; sudo pbuilder build <working-dir>/<project>_<version>.dsc
(02:41:37 PM) dholbach: done
(02:41:41 PM) dholbach: If the build succeeds, you can test-install the resulting package from /var/cache/pbuilder/result/.
(02:42:03 PM) dholbach: but that means that you need to have the packaging sorted out before, the tools won't "make that happen for you"
(02:42:10 PM) dholbach: on the plus side we have good info on how to get that done :)
(02:42:12 PM) dholbach: in the knowledge base
(02:42:17 PM) dholbach: <joaopinto> QUESTION: Do daily builds, PPAs, and official archives share the same infrastructure resources ?
(02:42:41 PM) dholbach: joaopinto: yes, it's all the same infrastructure, but build machines are assigned specific tasks
(02:42:50 PM) dholbach: and some builds have a higher priority than others
(02:43:23 PM) jcastro: ok so one last example
(02:43:31 PM) jcastro: (this is the OOOOH moment.
(02:43:49 PM) jcastro: so, in case you didn't know, some teams in ubuntu are already putting their packaging in bzr
(02:44:06 PM) jcastro: and eventually the entire distro will be in there
(02:44:10 PM) jcastro: so if we know where upstream is
(02:44:14 PM) jcastro: and where the packaging is ...
(02:44:26 PM) jcastro: we /should/ be able to make dailies out of almost anything!
(02:44:34 PM) jcastro: so ... I tried this:
(02:44:41 PM) jcastro: https://code.edge.launchpad.net/~jorge/+recipe/shotwell-daily
(02:44:54 PM) jcastro: (note the +recipe in the url, this is how we'll keep track of the recipes)
(02:45:01 PM) jcastro: shotwell is a photo editor
(02:45:06 PM) jcastro: I knew where upstream SVN is
(02:45:12 PM) jcastro: so I imported it into launchpad
(02:45:25 PM) jcastro: and I knew the ubuntu desktop team kept the packaging in bzr. So I tried to mush it together.
(02:45:45 PM) jcastro: and this is also where you see the "not finished" bits yet
(02:45:53 PM) jcastro: right now we don't have an easy way to do this, but it's a goal
(02:46:07 PM) jcastro: but from looking at that page you can get an idea of where we're going with this
(02:46:20 PM) jcastro: and yes, we're going to need more hardware. :)
(02:46:41 PM) dholbach: are there any more questions up until now?
(02:46:49 PM) jcastro: also, some tips here
(02:47:20 PM) jcastro: first off, we can't work in a vaccum, so while it may be fun to set up 50 dialy builds for your favorite project
(02:47:35 PM) jcastro: remember that they're not as useful if the upstream projects themselves don't know about it
(02:47:41 PM) jcastro: or know how to deal with it
(02:48:02 PM) jcastro: imagine if you were a developer and someone on the internet made daily builds available, and your bug tracker started getting alot of traffic
(02:48:08 PM) jcastro: and you weren't ready for it!
(02:48:28 PM) jcastro: so I encourage you not to just set these up, but work with app developers
(02:48:46 PM) jcastro: so that they can use these tools to make better releases and better software
(02:48:47 PM) dholbach: yeah, good thinking, plan with upstream
(02:48:58 PM) jcastro: more questions?
(02:48:59 PM) dholbach: talk to them, and go adopt-an-upstream
(02:49:17 PM) jcastro: yes, I personally plan on perfecting a few recipes locally
(02:49:24 PM) jcastro: and then mailing someone upstream
(02:49:25 PM) dholbach: (more info on that on Friday 16th, 18 UTC, same place :-D)
(02:49:39 PM) jcastro: and then have them set up their own based on my initial work, so that they can use them as they see fit
(02:50:36 PM) dholbach: more questions? or anything else you're wondering about?
(02:50:41 PM) jcastro: and of course, if you know projects interested in dailies, please let me know!
(02:51:25 PM) jcastro: I'm looking forward to seeing all the recipes people come up with
(02:51:55 PM) dholbach: yeah, me too
(02:52:01 PM) jcastro: we're working on listing some dailies here: https://wiki.ubuntu.com/DailyBuilds/AvailableDailyBuilds
(02:52:16 PM) jcastro: Since right now launchpad doesn't have a way of just showing them all at once
(02:52:25 PM) dholbach: maybe you have some suggestions or crazy ideas what you want to do with daily builds?
(02:52:46 PM) dholbach: and please: if you find bugs, file them, talk to the people in #launchpad about them, so we can iron all of them out
(02:52:51 PM) jcastro: oh, I forgot to mention
(02:52:56 PM) jcastro: this isn't just for desktop stuff
(02:53:13 PM) jcastro: chuck short has been offering dailies of mysql, puppet, and a ton of other server stuff
(02:53:35 PM) dholbach: <penguin42> QUESTION: Ah, there's a good question - if you find a bug in a daily, do you file it in the normal way?
(02:53:40 PM) jcastro: this is great for testing the entire platform you deploy on!
(02:54:10 PM) dholbach: penguin42: I think you can start with the launchpad-code project, but if bzr gives you an error message, or bzr-builder acts up, file the bug there
(02:54:17 PM) dholbach: if you're unsure, talk to the folks in #launchpad
(02:54:51 PM) dholbach: sorry, one thing we missed: you need to be part of the ubuntu beta testers team
(02:54:57 PM) dholbach: it's currently only available on edge
(02:54:59 PM) jcastro: https://edge.launchpad.net/~launchpad-beta-testers
(02:55:12 PM) jcastro: join that team if you want to test this feature!
(02:55:19 PM) dholbach: <penguin42> dholbach: I meant a bug in a particular daily built package, not in the daily-build mechanism
(02:55:27 PM) dholbach: sorry I misunderstood penguin42's question
(02:55:39 PM) dholbach: in that case it makes sense to file the bug upstream and let them know about it
(02:55:51 PM) jcastro: this is why it's important to talk to them
(02:56:01 PM) dholbach: apport for example will tell you that it's not an ubuntu package, but something else
(02:56:03 PM) jcastro: they might say "yeah, I know that's broken, don't file bugs"
(02:56:11 PM) jcastro: or they might appreciate a bunch of testing
(02:56:19 PM) dholbach: if you give out daily build packages to users, let them know what to do with bugs they found, give them detailed instructions
(02:56:40 PM) dholbach: if you haven't, bookmark https://help.launchpad.net/Packaging/SourceBuilds/GettingStarted :-)
(02:56:52 PM) dholbach: jcastro: I'm done - what about you?
(02:56:53 PM) jcastro: awesome, thanks for the tour daniel
(02:57:03 PM) jcastro: I am looking forward to seeing how app developers use this!
(02:57:12 PM) dholbach: thanks jcastro - these daily builds ROCK and will make life a lot more fun :)
(02:57:12 PM) jcastro: I think it'll go a long ways to getting fixes out to people much quicker!
(02:57:25 PM) jcastro: thanks everyone for participating!