LucidDailyBuilds

Summary

Daily Builds can be very useful for users, contributors, and distributions. Relying on them without due care could lead to very serious problems. However, with skilled maintenance and informed users the benefits can outweigh the risk.

We will work with the Launchpad developers to make daily builds easier to set up by building the functionality in to Launchpad. This will leave the maintainers more time to work on making their set of packages better and lower the barrier to entry.

In addition we will grow a community of those that are making daily builds where knowledge can be shared and best practices grown.

Release Note

None, as this is not something for end-user consumption.

Rationale

Daily builds can be immensely powerful. They massively lessen the barrier to using and testing code that is fresh from the fingers of the developers. They avoid you having to build a project from source every day, making sure to keep up with changes in dependencies. They allow you to be testing code almost as it is written, speeding up the feedback cycle to the developers, and potentially increasing the number of people involved in that feedback cycle.

They also allow you to verify bugs against the latest code, so that bug reports are of more relevance to the developers. If you so choose they can also be set up so that bugs are also tested with fewer distribution patches, further increasing the developers' confidence in the bug reports. This could be extremely valuable for Ubuntu developers and triagers, allowing them to move the bug reports upstream faster.

Having the more daily builds available for people to use if they wish allows more projects to get the benefits outlined above. While several projects have daily builds set up, it is still a huge minority of packages in Ubuntu that have any associated daily builds.

We want to build a community that knows how to put together a good daily build, that is useful for its purpose, and minimises the risks as much as possible. This community can then document their best practices and help others set up a daily build. They can also improve the tools, the packaging, and the projects to make daily builds easier and safer.

User stories

* Rob finds a bug in gwibber. He is asked to verify that the bug still exists in trunk, and so he installs the package from the PPA to test. Once he has verified whether the bug still exists he can choose to return to the distribution version, stay on the fresh version, or keep receiving daily builds.

* Michael is a big fan of the gnome-do package and he wants to contribute where he can. In order to do this more effectively he wants to run the new code that the developers are talking about so that he can help to test it. He subscribes to the daily build of gnome-do, so that whenever a developer adds a new feature he can be testing it the next day.

* Anne is a developer on KMail. She wishes to work on integrating some in-development features of the KDE libraries in to the app. In order to do this though she needs to track the development of the libraries she will be using very closely, so that the newest features and fixes for them are available to her and she is testing the latest code. To do this she subscribes to a PPA that contains the daily builds for all the libraries that she is interested in, and the libraries that they depend on. Every day she gets the latest code with an apt-get upgrade, rather than pulling from the VCS and building the libraries. This leaves her more time for working on the integration.

* Toby works on a package in Ubuntu. He sets up a daily build of the package so that he can get early warning of build failures when it is built on Ubuntu, dependency changes, and other things. This means that when a release is made he has less work to do to get the package ready for the distribution.

Design

Launchpad will implement daily builds based on bzr-builder's recipes.

Launchpad Features

Launchpad will be able to store recipes and then build the source package that they specify. This will make it easy to set up daily builds based on bzr branches. We will then rely on Launchpad's vcs-imports for handling other VCS. The full design and implementation will not be specified here as the Launchpad developers are currently doing that.

This may require some improvements to bzr-builder in order for them to be able to use it.

Community

We will work to grow a community of people interested in daily builds that can define their own policy on what a "good" daily build looks like. It is not anticipated that this community will be drawn mainly from the currrent Ubuntu development community.

The community will discuss and then document guidelines for setting up, maintaining, and distributing a daily build that others can follow in order to be reasonably assured that their build is maximally useful and minimally risky.

The team will look at things like upstream co-operation, emergency procedures, handling dependency build order, allowing users to get back to distribution packages, and version numbering.

Implementation

Community

http://launchpad.net/~dailydebs-team is a place where interested people can join to discuss daily builds. We will also create an IRC channel for interactive discussion.

We will hold IRC meetings to assess current progress and discuss issues we have found.

We will use DailyBuilds for documenting the things that we learnt.

Unresolved issues

* How to combine unrelated branches (currently lp:product and the corresponding lp:ubuntu/package)

* How to handle bugs from these packages. Currently there is no default place for them to go, and in particular apport will refuse to file bugs against a daily build package. This means that some of our most valuable issues can't be easily reported.

* How to sequence builds for strongly inter-connected packages.

  • - Either some way to block on the binaries of one being published - Or some way to manipulate Build-Depends to achieve the same.

BoF agenda and discussion

The goal is to produce packages of tip for a set of upstreams. Reasons for this:

  • bug triage against tip is easier
  • Make it easier for upstream to grow a community that use their latest code

To do this the Ubuntu patches are not too important, so just re-using the ubuntu packaging may not be the best thing to do.

Launchpad plans to provide a way to build daily packages from bzr-builder recipes which connect bzr branches.

bzr-builder is a bzr plugin that allows you to compose a set of bzr branches to get a desired tree.

Many times people will create a new packaging that doesn't have the patches.

Would be great to have packaging where we can selectively add in the "ubuntu sauce". Starting with the vanilla upstream would be the aim.

Ayatana used the recipes, with some infrastructure. Launchpad will provide this infrastructure for others. This will either be timed builds, on every commit, or when a tag is added.

As the history is not shared between the upstream branches and the packaging branches we currently have.

Stable branches are also interesting. A general framework will allow this to be done as well.

Some packages are very inter-dependent, so you need to sequence the builds.

  • If it is an API addition then the build will fail if the order is wrong.
    • - works itself the next day, so ok if it doesn't happen to much.
  • Robert's work allows us to block until the binaries are published, which
    • might help.
  • Otherwise using the Build-Depends would work.

Failures need to be communicated to the right people in the right way.

Are they fragile from experience?

  • - Not too much. - Some projects may have lots of build failures. This feedback might help increase
    • their quality. - Building on commit for these projects wouldn't be a good idea.

Will have inheritance one day. UI for LP will be interesting.

<smagoun> Will the PPA buildd pool be expanded to handle the increase in daily builds? Currently just the browser dailies (chromium, firefox) are enough to cause significant queues in the PPAs.

  • - Yes

<qense> Maybe a nice addition: on-demand building of Bazaar branches that fix bugs, allowing you to add ppa:ubuntu/lucid/bug123456 and install the fix to test it? +1

  • - Launchpad's branch selector could allow you to find these branches - Maybe based on some flag of the branch? No real sense in packaging something that's still in flux and isn't at a stable state.

A community will grow, and one person may be able to cover 10-20 upstreams, as they are not generally going to be modifying the code.

  • - Should have a way for this community to communicate, and some guidelines/tutorials
    • to go with it.

Ubuntu One are keen to do this.

Aborting builds will be important if we have lots and lots of packages built daily.

Launchpad's plans:

  • buildds need some work to generalise them from "build package" to "run something" - currently underway
  • build from recipe to archive will be landing in the next 6 months.
  • UI to do it needs to be worked on.
  • Also need to discuss how to link it in with the rest of LP.

Building a community around it:

  • dailydebs-team is available, or we could use ubuntu-distributed-devel: jml and james_w to discuss.
  • IRC meetings to discuss progress and get feedback: james_w
  • IRC channel on freenode: james_w
  • wiki.ubuntu.com/DailyBuilds for documenting

Bazaar are working on importing more upstreams.

  • - Regularly enough? Every 6th hours. - Cross reference lists of top 100 bug packages with code imports.

CategorySpec

FoundationsTeam/Specs/LucidDailyBuilds (last edited 2009-12-03 04:15:50 by 74)