GrumpyGroundhog

Revision 6 as of 2005-04-26 01:54:51

Clear message

The Grumpy Groundhog

Status

Introduction

The Grumpy Groundhog Project aims to produce a crack-of-the-day set of packages, in a release called "grumpy". Grumpy will never actually be released, it will be in a state of perpetual development, representing the very cutting edge of upstream and distro packaging. The packages in "grumpy" will be produced automatically by combining the very latest daily upstream (from CVS, SVN, Baz, BK or Arch) with the latest distro patches (from the current development release of Ubuntu, or specifically produced for grumpy to address build issues).

Rationale

Upstream development in the open source world moves at a tremendous pace. Many developers like to keep up to date with specific upstream products, but the work involved in building from CVS every day is substantial. By offering the Grumpy Groundhog release, Ubuntu provides those developers with a ready source of daily built packages containing the very latest upstream code.

Regular and continuous building of new upstream code will allow cutting edge developers to keep track of changes in the upstream codebase that might affect the distribution in due course. For example, since these packages will be auto-built with the very latest compiler code, failures in the build cycle may indicate changes that are required to keep upstream compatible with the next release of the build toolchain. Should we make grumpy available on multiple architectures, we would provide an early warning system of porting issues with new daily upstream code.

Scope and Use Cases

Grumpy is designed to be a snapshot of every package in Ubuntu built against its crack-of-the-day upstream code.

  1. Frank is an Ubuntu developer, and particularly interested in the Apache
    • package. He would like the apache2 package to be part of Grumpy, so he can monitor daily snapshot builds of the package. He notices that Apache2 upstream is being published in Baz format in the Launchpad, so he sets up Apache2 for building in Grumpy. Every day, Launchpad attempts to build an apache2 package using the latest upstream SVN code (as imported and published in Baz format) together with the current set of patches in use for Grumpy. If the package successfully builds, it is published in grumpy.
  2. Harry is an end-user of Ubuntu, not a hard-core developer but
    • nevertheless interested in what's going on in the Apache project. He uses the latest stable Ubuntu release for his desktop, but out of curiousity he has installed the apache2 package from Grumpy. His apt sources have Grumpy pinned low except for apache2, so the only package on his system from Grumpy is apache2, everything else is from Hoary. On a regular basis he updates his system and gets the latest apache2 package with crack-of-the-day upstream code. The version of the package includes a build date identifier, so he knows when the package was built.
  3. William is interested in whether or not the upstream apache2 code will
    • build with the latest GCC, so he subscribes to the package build notifications for apache2 in Grumpy. Every day, when Launchpad assembles and builds the latest apache2 Grumpy package, he receives a log of the results of the process. He will see the results of the attempt to produce the source package, and any failures patching the upstream source with the current set of grumpy patches. If the source package is successfully constituted he also sees the log of the build attempt, and an indicator of its success or failure.
  4. Frank notices that apache2 did not build in Grumpy today. He looks over
    • the log and sees that one of the distro patches failed to apply, so he updates that patch using Baz and HCT to resolve a conflict introduced by yesterday's upstream commits. He satisfies himself that the updated package builds for him locally, and commits the updated patch into the Grumpy repository. Tomorrow, the package will likely build correctly.

Implementation Plan

The basic implementation is that the upstream release will be mirrored from CVS or something similar into a baz branch. That branch is then merged with others representing packaging patches from Debian and Ubuntu. The resulting tree is built and uploaded to a repository.

In many cases of course the software will fail to build on at least some systems, or fail to package, or be uninstallable or unusable. There should be an interface for reporting those failures to interested parties, including the upstream maintainer and the packager. Perhaps a view of the cross-platform build similar to http://build.samba.org/ would be useful. When the patches fail to apply that information should be fed back. Note that many of these problems will be transient and go away in the next CVS update, so the failures should be unobtrusive and should not persist once they are fixed. Perhaps there should be an option to open a Malone bug for the failure but it should not be done by default.

This is an extension of the existing HCT build system. Rather than starting the branch from the last release tag we instead run the process from the very top of CVS HEAD.

Since the code at the top of HEAD is even less trusted than regular releases we have to be careful to build it in a protected chroot, possibly eventually in a virtual machine.

Each build should be marked with unambigious identification of the baz revision from which it was built. Since this may not mean anything to the upstream maintainer we should also identify which cvs/svn version was used as the basis. For CVS there is no global identification so the time and last commit message may be enough.

Packages Affected

All.

User Interface Requirements

Combining upstream branches with branches extracted from other distributions will produce an enormous number of branches and revisions. This may stress the VCS but also makes it difficult to present the changes to a user. Consider trying to display the SuSE kernel which has applied a thousand patches, each of which may have its own history. On 1st August there should be a thousand CVS branches syncing in to baz.

The UI may wish to distinguish major "trunks" of development (large branches) so as not to flood the user with details on all teh small patches. So the Debian branch will be shown in terms of which patch branches are in or out compared to Ubuntu, and then the user may drill down to see the details.

Outstanding Issues

  • What social issues might arise when we bring Grumpy on stream? Might it be hard to get developers focused on the current frozen release when they can get their fix of daily crack from Grumpy? Might some people want to try and run end-to-end grumpy, or is nobody actually that extreme?
  • Some upstream developers might be grumpy about us shipping probably-broken development code. In particular some foolish users might run grumpy builds and suffer data loss bugs, causing either bogus bug reports or loss of reputation for the package. We possibly need an opt-out for packages that don't want to be built this way. GNU emacs is reported to be grumpy about people shipping pre-release packages. We need to communicate well and be sensitive to their requests.
  • On the other hand, some packages (mplayer?) encourage people to build from CVS HEAD. These might be good demonstrations of grumpy power.
  • How do we deal with build tool updates? Should we attempt to rebuild every package in Grumpy when we update GCC? Should we attempt to build every package every day with the latest GCC?
  • Should grumpy have packages for unstable branches that are not actually part of the current development distrorelease? For example, should it have a gimp2.5 package if the current stable distrorelease just has gimp2.4? Or should it have both gimp2.4 (updated for any new patches to the upstream stable release) as well as gimp2.5 (representing the latest upstream unstable branch)?
  • When upstream projects use Bazaar-NG, we will be able to identify when a patch has been merged upstream and no longer treat it as a distribution patch. Patches might be taken in through a cherry-picking operation.