Archive rebuilds in the cloud for armel

• Launchpad Entry: mobile-karmic-arm-cloud-builds.

• Created: 2009/04/29

• Contributors: Michael Casadevall, Loic Minier

• Packages affected:

Summary

Creation of a tool to build packages for armel in the cloud, allowing optimized binaries with different toolchains, QA, and perhaps PPAs.

Release Note

Ubuntu now provides a framework for doing armel rebuilds in the cloud.

Rationale

• Creation of optimized binaries for OEM
• Testing of compiler flags to determine what we want to enable/disable
• Possibly used in implementation for ARM PPA
• Also can be used to add builds for older ARM architecture variants

User stories

Assumptions

Design

• Based around an EC2 API (either EC2 itself, or private cloud)
• One build master and build daemons, with control point being in the cloud.
  • - Want to be able to use the subset between EC2 and Eucalyptus
• build around the standard debian build tools (sbuild)
• For wanna-build database with current debian wanna-build, or wb_python.
  • - Debian buildd system is email based - Being able to easy track and (if necessary) import new sources; perhaps
    • based on quinn-diff
• Input is a list of APT sources
• Archive software
  • mini-dak, dpkg-scanpackages, apt-ftparchive?
  • needs to preserve components
  • Backend? filesystem? Should be in S3
• Multiple passes? Using binaries from the old or currently built archive?
  • Probably an option
  • Pass for required + base + build-essential packages? Check with infinity
• Keep build logs
  • Useful to debug things if you do a second pass
  • Maybe base around wanna-persue
• Command-line with a config file
• Optionally: import the archive into the cloud to save bandwidth

Implementation

UI Changes

• No UI changes, new framework being written.
• New UI to likely be command-line based
• wanna-build/buildd should be modified not to use email if possible
• wanna-persue (if used for web interface) should have a summary view added to it

Code Changes

• Creation of an rebuild-driver which handles all the implementation steps above
  • Creates w-b databases for each stage of the rebuild
    • Stage 0 - rebuild of base system
    • Stage 1 - first pass of entire system
    • Stage 2 - rebuild of the entire system to ensure sane end result (optional)
• buildd should be modified to be able to upload unsigned packages assuming success
• Entire framework should have a turn-key like setup

Migration

• No migration, new functionality

Test/Demo Plan

• The rebuilder should be tested by doing a full archive build, installing the result, and making sure the end results work

Unresolved issues

• How to handle blacklisting of packages (i.e., taking a package from a local source, and confirming it always superseeds any other package.)

---

CategorySpec