Ubuntu Wiki

Dev Week -- Hacking Soyuz to get your builds done -- noodles775, cprov and wgrant -- Fri Sep 4th, 2009

(04:00:53 PM) noodles775: My name's Michael Nelson and I've been working on Launchpad and Soyuz for around 9 months now.
(04:01:17 PM) noodles775: Here's an overview of what's coming up over the next 40mins or so:
(04:01:26 PM) noodles775: 1. Grill a new soyuz hacker with questions.
(04:01:34 PM) noodles775: 2. A guided tour through the Soyuz code-base
(04:01:47 PM) noodles775: 3. Setting up a Soyuz test scenari
(04:01:57 PM) noodles775: o
(04:02:17 PM) noodles775: So - up first is our chance to grill the latest Soyuz hacker: wgrant! Since the open-sourcing of Soyuz with Launchpad, wgrant has - in his own time - pushed 20 (!) launchpad branches.
(04:02:34 PM) noodles775: 7 of these are soyuz-related branches (afaics).
(04:02:45 PM) noodles775: wgrant: wanna introduce yourself?
(04:02:53 PM) wgrant: I didn't think I'd done quite that many, but OK!
(04:03:12 PM) wgrant: So, I've been an Ubuntu developer for a few years now.
(04:03:16 PM) noodles775: (that's how many I'd counted that have been pushed - not necessarily merged :) ).
(04:03:29 PM) wgrant: And at some point became interested in the infrastructure behind it all.
(04:03:45 PM) noodles775: Cool!
(04:03:48 PM) noodles775: So this is our chance to find out how William got started working on Launchpad and Soyuz, what the issues were, and what he'd recommend to others who want to hack on Soyuz.
(04:03:55 PM) wgrant: So when it was sneakily open-sourced a month ago, I jumped straight in.
(04:04:31 PM) wgrant: There are lots of bits and pieces I'd like to see fixed or improved, so it was really great to see the source releaed.
(04:04:54 PM) wgrant: Hacking Soyuz (and Launchpad in general) is probably going to be a little intimidating at first.
(04:05:14 PM) wgrant: But Launchpad developers are very helpful to new contributors, so they can give you a lot of guidance if you get lost.
(04:05:30 PM) noodles775: BTW everyone: please have think up some good questions that will help you get started and post them in #ubuntu-classroom-chat
(04:05:48 PM) wgrant: To get oriented for development, I'd start off by setting up a local development environment (https://dev.launchpad.net/Getting).
(04:06:14 PM) wgrant: Then have a look at how the codebase is organised. Maybe poke around in the model a bit with 'make harness'.
(04:06:28 PM) wgrant: See how things work using the SoyuzTestPublisher, which I believe noodles775 will explain later.
(04:06:49 PM) wgrant: Once you've found your way around a bit, identify a little bug or feature on which you want to work.
(04:07:32 PM) wgrant: The next step is to ask a Launchpad developer (in #launchpad-dev) about it. They'll advise you whether you're attempting the impossible, or otherwise tell you where to start.
(04:08:17 PM) wgrant: That bit is quite important, as it can stop you from hitting dead-ends or attempting something that's just too difficult for a first-time hacker (as quite a few things are).
(04:10:52 PM) noodles775: cprov: have you (or anyone else) ever tried to explain what soyuz does by analogy?
(04:10:56 PM) wgrant: An analogy... a good question. It's a pretty complex creature, so I'm not sure where to start.
(04:11:59 PM) noodles775: I've sometimes tried to think of what soyuz does as a blogging engine... something familiar, with some similarities...
(04:12:20 PM) noodles775: But there are quite a few differences too.
(04:12:30 PM) noodles775: OK, any other questions for wgrant ?
(04:12:44 PM) cprov: well, Soyuz encompasses a lot of subcomponents that takes debian source packages as input and produces debian repositories, but there is a lot of details in the middle.
(04:13:16 PM) noodles775: Which is a great lead-in to 2. A guided tour through the Soyuz code-base - take it away cprov :)
(04:14:18 PM) cprov: Hi, my name is Celso Providelo and I've been working on Soyuz for the last 5 years (!)
(04:15:08 PM) cprov: so, I would like to point you to some piece of documentation I've created to guide users to the Soyuz code base.
(04:15:17 PM) cprov: https://wiki.ubuntu.com/CelsoProvidelo/SoyuzInfrastructureOverview
(04:16:07 PM) cprov: it has a descent (but not pretty) diagram, it illustrate what I mean by 'lots of details in the middle' before.
(04:17:15 PM) cprov: Soyuz is in reality a set of integrated tools/components for 'controlling' software packages.
(04:18:25 PM) cprov: It starts with the 'upload server', an FTP daemon that receives source packages uploaded by users using `dput/dupload`.
(04:19:43 PM) cprov: Sources are them passed to the 'upload processor' which verifies their consistency (packaging metadata) and stores its information in the Launchpad database.
(04:20:46 PM) cprov: the publication of the source automatically creates a build request, which is dealt by the 'build dispatching' component.
(04:21:47 PM) cprov: it pass the source to a 'builder', an isolated environment for running `debuild`.
(04:22:46 PM) cprov: Binaries resulted from the build process come back to the upload processor and are checked before getting stored in Launchpad.
(04:23:07 PM) cprov: QUESTION: c_korn: cprov: those are the builders ? https://launchpad.net/builders
(04:23:23 PM) cprov: c_korn: exactly, those are the current build machines.
(04:24:13 PM) cprov: In the wiki page I've mentioned above, there are pointers to the corresponding modules for each part of the systems
(04:25:50 PM) cprov: knowing the topology described in that diagram and where to look in the codebase will help you to find out what need has to be changed.
(04:27:30 PM) cprov: That document is very short on details, but I expect them to be added as long as we get more community contributors. Don't hesitate in adding questions or suggestion directly to it.
(04:28:00 PM) cprov: noodles775: the stage is yours, I guess.
(04:28:17 PM) noodles775: cprov: I had a few questions myself...
(04:28:30 PM) noodles775: <noodles775> QUESTION: What's an example of a inconsistency that the upload processor will find and reject?
(04:28:58 PM) cprov: noodles775: basically anything that doesn't pass a `lintian` check.
(04:29:19 PM) noodles775: ah ok, so it's really just to check for things that people should do themselves before uploading.
(04:29:30 PM) cprov: exactly
(04:30:07 PM) noodles775: OK, on to: 3. Setting up a Soyuz test scenario
(04:30:07 PM) cprov: the upload processor also checks consistency against the packages previously uploaded
(04:30:46 PM) noodles775: When hacking on Launchpad?s soyuz application ? and creating tests to verify that your new functionality works, you?ll often need sources or binaries published in very specific scenarios.
(04:31:09 PM) noodles775: We're going to use a special test feature - the SoyuzTestPublisher - to publish sources and binaries to a PPA in our development environment - and watch the status' update live in the browser.
(04:31:26 PM) noodles775: The SoyuzTestPublisher ? as the name suggests ? was created for this exact reason (by cprov) :)
(04:32:03 PM) noodles775: So for this hands-on - you don't need any previous LP development experience... but you do need a Launchpad development setup.
(04:32:20 PM) noodles775: If you've set up the Launchpad development environment properly according to http://dev.launchpad.net/Getting, you should be able to run the following command:
(04:32:43 PM) noodles775: $ rocketfuel-branch soyuz-test-scenario
(04:33:32 PM) noodles775: While that's going - can I get an idea of how many (if any) people are following along?
(04:34:26 PM) noodles775: Great! As long as there's at least one person, it's worth doing :)
(04:34:34 PM) noodles775: When that's finished, change into the soyuz-test-scenario directory.
(04:34:55 PM) noodles775: We will be watching the new publications at:
(04:35:03 PM) noodles775: https://launchpad.dev/~cprov/+archive/ppa
(04:35:22 PM) noodles775: This page updates the build status column every 60 seconds by default, so instead of tapping your fingers while you wait I'd recommend specifying an update interval of 5 seconds for the dynamic updates
(04:35:22 PM) noodles775: as shown in the patch: http://pastebin.ubuntu.com/264831/
(04:35:41 PM) noodles775: You can apply that patch to your current branch with:
(04:35:53 PM) noodles775: $ wget http://pastebin.ubuntu.com/264831/plain/ -O update_every_five.diff
(04:35:53 PM) noodles775: $ bzr patch update_every_five.diff
(04:36:58 PM) noodles775: Just let me know if I go too fast...
(04:37:01 PM) noodles775: With that patch applied, run 'make run' in your branch directory in one terminal and 'make harness' to get a python console in another.
(04:37:34 PM) noodles775: Now, using the python console, we'll first just grab a sample-data user who has a PPA.
(04:37:47 PM) noodles775: >>> cprov = getUtility(IPersonSet).getByName('cprov')
(04:38:15 PM) noodles775: A few imports that we need
(04:38:23 PM) noodles775: >>> from lp.soyuz.tests.test_publishing import SoyuzTestPublisher
(04:38:23 PM) noodles775: >>> from lp.soyuz.interfaces.publishing import PackagePublishingStatus
(04:40:14 PM) noodles775: and then we create our Soyuz test publisher instance:
(04:40:26 PM) noodles775: >>> publisher = SoyuzTestPublisher()
(04:41:11 PM) noodles775: Next, we just to ensure that the publisher has default distroseries etc. setup:
(04:41:23 PM) noodles775: >>> publisher.prepareBreezyAutotest()
(04:41:57 PM) noodles775: And now for the fun, we'll create a new published source package:
(04:42:05 PM) noodles775: >>> testsrc = publisher.getPubSource(sourcename='testsrc', archive=cprov.archive, status=PackagePublishingStatus.PUBLISHED)
(04:42:31 PM) noodles775: Finally, we'll create the missing builds for this new source package, and commit it all to the db:
(04:42:40 PM) noodles775: >>> builds = testsrc.createMissingBuilds()
(04:42:40 PM) noodles775: >>> import transaction;transaction.commit()
(04:42:56 PM) noodles775: Now open a browser at https://launchpad.dev/~cprov/+archive/ppa (or re-load) and you'll see the new 'testsrc' package with its pending builds.
(04:43:19 PM) noodles775: We'll now update the build manually watching the status update itself in the browser window.
(04:43:30 PM) noodles775: >>> from lp.soyuz.interfaces.build import BuildStatus
(04:43:41 PM) noodles775: >>> build = builds[0]
(04:43:48 PM) noodles775: >>> build.buildstate = BuildStatus.BUILDING
(04:44:05 PM) noodles775: Just watch your browser window without refreshing... after you commit the transaction, you'll see the build status for your package update within 5 seconds:
(04:44:17 PM) noodles775: >>> transaction.commit()
(04:45:09 PM) noodles775: Did it work?
(04:45:33 PM) noodles775: Now we update it to fully-built:
(04:46:17 PM) noodles775: >>> build.buildstate = BuildStatus.FULLYBUILT
(04:46:27 PM) noodles775: >>> transaction.commit()
(04:47:33 PM) noodles775: Now we've got a successful build, but its binary has not been published,
(04:47:50 PM) noodles775: Mouse-over the build icon to see a description of the current state.
(04:48:20 PM) noodles775: So we'll fake the successful publication of the binary with the SoyuzTestPublisher...
(04:48:30 PM) noodles775: >>> binary_pkg_release = publisher.uploadBinaryForBuild(build, 'testbin')
(04:48:45 PM) noodles775: >>> binary_pub = publisher.publishBinaryInArchive(binary_pkg_release, cprov.archive, status=PackagePublishingStatus.PUBLISHED)
(04:48:58 PM) noodles775: Again, be ready to watch it update:
(04:49:10 PM) noodles775: >>> transaction.commit()
(04:49:36 PM) noodles775: There you go! A brief intro to the SoyuzTestPublisher for testing soyuz publications.
(04:49:54 PM) noodles775: I've created a screencast and paste of the script at:
(04:50:06 PM) noodles775: http://micknelson.wordpress.com/2009/09/04/testing-launchpad-soyuz-features/
(04:51:01 PM) noodles775: So, that's all we had... does anyone have any questions?
(04:52:24 PM) noodles775: I guess not :) Well, hope it was useful! Remember, if you've got any questions later, you can always ask them on #launchpad-dev.
(04:55:36 PM) noodles775: jcastro: ?
(04:55:43 PM) noodles775: <c_korn> some final words to end the Ubuntu Developer Week ?
(04:58:36 PM) jcastro: not really
(04:58:40 PM) jcastro: see you guys next cycle? :)
(04:58:50 PM) jcastro: \o/
(04:58:52 PM) noodles775: :)
(04:59:19 PM) jcastro: we'll have an open week coming up soon so there will be more tutorials, etc.
(05:01:26 PM) jcastro: please feel free to send your feedback to myself or daniel holbach

Outline

Here's an outline of what we can expect during the Hacking Soyuz session - but like most UDW sessions it will be flexible and might stray with the right questions

Grill a new soyuz hacker

OK, so William isn't exactly new to Soyuz - but he has been Soyuz's biggest community contributor since the open-sourcing. Being a relatively new soyuz hacker myself (noodles), I was keen to hear William's thoughts and responses to the following questions: (if you've got questions for William too, ask them in #ubuntu-classroom-chat too!)

Question 1

How did you originally get involved with Soyuz - and why do you stay involved?

Question 2

If you wanted to describe Soyuz to an Ubuntu/Debian user, what analogy would you use and why? (It could even be worth seeing if any participants have analogies too - exploring where each is good and where each fails? Just a thought).

Question 3

If a person came to you who was competent in Python, had some packaging experience with their Launchpad PPA and was really keen to hack on Soyuz, how would you recommend they get started?

Perhaps leading into a topic/segment that wgrant would like to lead?

A tour of the Soyuz code-base

Celso's guided tour of code, documentation, API's and concepts.

Setting up a Soyuz test scenario

For this section, you'll need a working Launchpad development environment. We'll be using:

• make run - to view our soyuz publications
• make harness - to publish new packages and simulate builds completing so that we can view them in our browser.

See Testing launchpad soyuz features for the screencast and code snippets.

How can I get involved / General questions

What are the plans for PPA/Desktop integration, different strategies people have adopted, pros/cons.

What are the plans for the soyuz-upload handling.