Summary

Packages that represent the latest code from a project lower the barrier to contribution to the project. They allow users and distribution developers to follow the latest code without having to build themselves from Bazaar.

Vanilla packages of the latest code also provides an opportunity for confirming of bugs in conditions that will be most useful to upstream. This can ease the flow of bugs from distributions to upstream.

Making it easy for people to set-up daily builds of the latest code will lead to more projects offering them.

Release Note

This doesn't really have a place in the release notes as it doesn't change the distribution as such.

Rationale

Testing the HEAD of upstream's VCS can be very valuable, but building it is not easy for users, and is tedious to do regularly. Automatically producing these packages every day will alleviate some of this, but we would not have the resources to do this for every package currently. In order to investigate the complexities involved in doing it we will select a few packages and build a proof-of-concept that produces daily packages for them.

Use Cases

Assumptions

Design

The tool will work based on Bazaar branches, so that it can take advantage of bzrlib to do the heavy lifting.

A "recipe" will specify the parts that go together to make up the source, including the addition of the packaging, and any patches that need to be applied. It will be a fairly simple format, specifying branches to merge (and optionally specific revisions of them). There will be a second permitted operation, "nesting" that will allow composite packages to be assembled from different branches.

The tool will take a recipe file as input, and will follow the steps that it specifies, producing a source package, which will then be optionally signed and uploaded to a PPA. This then allows the command to be placed in cron to run every day, with modifications done to the recipe, or to the branches it specifies as needed.

As part of its operation the tool should generate a "manifest" and store it in the source packages. The manifest will show the exact revisions used from each branch, so that the package can be recreated later.

There are a few other pieces of information that are needed at certain points in the operation of the tool, for instance the GPG key id to sign the package if that stage is needed. This information could live in the recipe, or in a separate info file.

Often there will be no need to build the package if none of the branches have changed, but sometimes it may be desired to do this anyway, so that any problems caused by changes in the dependencies can be caught. There should be a way to specify which behaviour is wanted for each build.

In order to provide the facility to not build the package if none of the branches have changed then the tool needs to have some way to know what revisions the branches were at in the previous build. Copies of the branches that were used in the last build would allow you to do this easily, however they could take up a significant amount of diskspace, and would constrain you to always building on the same machine. The alternative is to then keep around, or make available, the revisions that were used in the last build.

The tool must generate at least part of the version number such that they increment on every build. Usually a base version will be specified, such as the version number that upstream is working towards in the branch that is being followed. The tool will then append to this the generated part. For daily builds the current date could be used, but it would be good to support multiple builds in a day for faster moving projects. In these cases some combination of the date and the revision number of each branch could be used, as long as builds weren't done if nothing has changed. Alternatively the time could be included, as well as the date, this would work for any scheme.

Some projects will require certain commands to be run at certain points in the build, e.g. autoreconf. There are a few possible solutions to this. Firstly, this could be a branch that is merged, but this would quickly get out of date and so cause build failures, which could be automatically solved. Secondly the package could be changed to do this at build time instead, but this may not work for every use case. Thirdly, we could support commands in the recipe, which will be executed as the recipe is processed. This does allow arbitrary code execution, but this can already be done in the clean target of debian/rules.

Implementation

The recipe will be a simple text file, for example 

branch lp:firefox
merge lp:~fta/firefox/fix-build
merge lp:~mozillateam/firefox/ubuntu.jaunty
merge lp:~asac/firefox/fix-installation
branch lp:xulrunner xulrunner
merge lp:~asac/xulrunner/embed

where there are two operations, "branch" and "merge". It reads and works similar to a list of bzr commands, but there are some small differences. The main one is that "merge" is done in to the most recent "branch" target, not in to the root.

This is a more complex example. A basic recipe for most packages would just be 

branch lp:project
merge lp:ubuntu/jaunty/project

with the more complex example showing how fixes can be applied where needed for the daily build.

The extra "branch" in the example shows how nesting can be used. In this case it embeds a copy of xulrunner in to the package, and then merges a branch that makes it suitable for embedding. This should be rarely used, but allows for more packages to be done in such a scheme.

Specific revisions of the branches can be specified like so: 

branch lp:project
merge lp:ubuntu/jaunty/project -r<revision>

where "revision" is any bzr revision specifier.

The manifest will then simply be a recipe with revisions specified. For instance running the tool with the the second example may lead to the following manifest. 

branch lp:project -rrevid:foo@bar.com-12345
merge lp:ubuntu/jaunty/project -rrevid:fta@ubuntu.com-q2434fr

This means that the manifest can then be fed back in to the tool to reproduce the original steps.

If given the the manifest of the previous run the tool can also detect whether there are changes in any of the branches. The manifest could either be stored somewhere known, or the tool could possibly extract it from the source package if it knows where to find it (e.g. in a PPA).

Version numbers

Most version numbers will be constructed from a base version number, with a generated part appended. A simple way to do this would be to have the base version specified in the recipe. 

base-version 5.1~

with the generated part being 

200902111429-1

when these are concatenated you would end up with 

5.1~200902111429-1

It would be possible to extend this to include information about the revision numbers of the various branches, but the above scheme should function well enough.

Supporting commands

Adding another possible operation to the recipe, "command" would allow extra commands to be run at certain points, e.g. 

branch lp:project
command autoreconf -f -i -v
merge lp:ubuntu/jaunty/project

The command would be run in the target directory of the last "branch" operation, and would be passed through the shell, allowing more flexibility.

This will not be supported. Allowing commands to be run makes the results of the tool unreproducible. The manifest allows you to reproduce the parts from combining branches, but running commands would break that.

Any commands that need to be run should be run at build time, even if this means modifying the packaging.

Test/Demo Plan

In order to test the tool a few daily builds will be set-up and maintained. In addition to making sure the tool works well it will also show how much work is required to maintain daily builds, which is interesting information.

There is no need to have a contingency plan, as this work does not affect the distribution itself. The earliest the tool itself would end up in Ubuntu is in Juanty+1, and the daily-built packages won't go in to the repositories.

Unresolved issues

* Is there extra information needed about the recipe than just the steps, e.g. desired source package name, GPG key to sign with? Should this live in the recipe somewhere? * Is the described recipe format flexible enough? Is it possible to unambiguously talk about a location in the file to add extra steps?

Gobby Notes from UDS Karmic Session 1

'Expanding and Socializing the Daily Builds Concept & Capability'

https://blueprints.edge.launchpad.net/ubuntu/+spec/foundations-karmic-daily-builds

CategorySpec

DailyUpstreamBuildsPOCSpec (last edited 2009-10-26 13:14:21 by eth0)