ImageBuildPipeline

Summary

Shorten the time required to get from a developer source upload to a full set of updated image builds on all architectures.

Release Note

This feature has no end-user impact.

Rationale

When preparing milestone releases, we often need to turn around fixes quickly. The pipeline from a developer source upload to a full set of updated image builds on all architectures is currently somewhere in the region of nine hours. We would like to make this much quicker.

Design

Publishing

The publisher currently runs Germinate 40 times to compute Task fields and the like. This is a necessary operation, but it could be much more efficient. We will expose a Python interface to Germinate for Launchpad to use directly, and arrange for it to involve much less repeated work; at the very least, Packages and Sources files are being scanned eight times as often as they need to be, and likewise the core parts of the seed structure are being expanded eight times as often as they need to be.

As a stretch goal, given this work it should be possible to have Germinate operate on the to-be-published state of the archive rather than on the just-published state, thereby eliminating a very long-standing source of hysteresis: certain changes require two publisher runs to settle properly.

We discussed an option to disable Germinate for certain quick runs, but it should not be necessary after this work.

We discussed switching to "no-more-apt-ftparchive", the publisher scheme used for PPAs. This does not currently support all the fine details of the primary archive, and would need more work; it is also not obvious that it would necessarily speed up publishing of the primary archive. We have no objection to further work in this area but it is not a priority.

Live filesystem building

Live filesystems are currently built using manual ssh triggers to non-Launchpad builders. This has been OK in the past, but it's failing to scale to the demands of ARM: we need to be able to distribute builds to a larger number of machines, and for that we need a proper scheduler, which might as well be Launchpad. Adam Conrad plans to work on making live filesystem builds be just another type of Launchpad build job, storing a single output at a time in the librarian.

We may need to implement builder affinity, since we may wish to preferentially allocate live filesystem build jobs to the fastest machines, at least around release time.

Live filesystem building on some architectures is sufficiently I/O-bound that it would benefit from qemu. We will look into this further if it looks as though fast enough ARM hardware won't be available in time.

Mirroring

The cdimage master system keeps a local mirror of the archive, partly for performance and partly so that it can make hard links into it for debian-cd. This mirror must remain under the control of cdimage, since updating it in the middle of a build is likely to cause failures.

There have been some implementation problems with this scheme. At the 11.10 release sprint we discovered pathological behaviour in the mirror script, dating from the very early days of Ubuntu when inter-component package moves were handled differently, and we fixed that for a very substantial performance gain. In future it may be worth investigating a move to lmirror, but this needs to be deployed on ftpmaster first.

The mirror is currently fetched from syncproxy or ports depending on the architecture. We will consolidate this to fetch directly from ftpmaster; this will cut out any delays caused by needing to wait for syncproxy/ports to catch up.

Everything else

The current cdimage master system is very heavily I/O-bound, with one of the highest rates of unfulfilled I/O in the datacentre. IS intends to move this to the new SAN, probably the week after UDS. This should make a significant difference on its own, both to mirroring and to other miscellaneous parts of the process. After this happens we will do more controlled testing of mirroring performance.

On the old system, timings of the cdimage master part of typical image builds look like this:

  • Mirror sync at start: 1-3 minutes
  • Germinate: 2-3 minutes
  • For live images:
    • Downloading live filesystem images: 1 minute
    • Assembling ISOs: 4 minutes
  • For d-i images:
    • Assembling ISOs: 10 minutes
  • Publishing (including making torrent metafiles etc.): 1.5 minutes

This has been optimised in the past and there is probably not a great deal of low-hanging fruit left; it also parallelises reasonably well.

We tangentially discussed moving Wubi into the archive properly rather than fetching it from Evan's home directory. This might well be a good idea for other reasons, but is out of scope for this specification since it would probably actually slow down the process of delivering Wubi changes!

cdimage is now able to feed output to the ISO testing system, provided that CDIMAGE_POST_QA=1 is set in the environment. This reduces the need for human interaction.


CategorySpec

FoundationsTeam/Specs/ImageBuildPipeline (last edited 2011-11-09 15:54:51 by cjwatson)