ARMAutomatedBootstrap

Summary

Develop a mechanism to break build-dependencies loops to rebuild Debian archives from scratch.

Release Note

If we have ARM specific release notes: a subset of the source packages may now be rebuilt from scratch.

Rationale

To truly rebuild repositories from scratch (e.g. using a new toolchain), or to bootstrap new ports, it would be desirable to have a mechanism to break build-dependencies loops and rebuild a full Debian archive.

User stories

Peter would like to bootstrap a new Ubuntu architecture for his new silicon, but all the packages build-depend on each other; with extended build-deps information, he may now break the build-dep loops in the proper places.

Brian would like to rebuild the whole Ubuntu archive with a new toolchain, ensuring that the results don't have any bits built by the previous toolchain; he starts a build from scratch using extended build-deps information to make sure that he builds the packages in proper order.

Assumptions

Design

Extend source packages meta-information to allow for multiple "stages".

Build-Depends-StageN fields could describe the build-deps needed for the various stages, e.g. Build-Depends-Stage1:, Build-Depends-Stage2: etc.

dpkg-buildpackage would pass DEB_STAGE in the environment as to allow changing debian/rules behavior based on the stage.

All possible packages (even those specific to early stages) would be listed in control, but the build would only output a subset, depending on the current stage.

Implementation

Start with a sbuild based prototype on a subset of packages and provide some sample modified packages and a modified sbuild to set the stage. Perhaps a modified wanna-build or another scheduler to demo the rebuild from scratch of these packages.

Test/Demo Plan

Run prototype against set of source packages to rebuild them from scratch.

Unresolved issues

Bootstrapping of build-essential might be harder, especially toolchain. In the typical cross-case, a cross-toolchain is needed which is a slightly different problem.

BoF agenda and discussion

Sometimes want to build archives from scratch (e.g. new architecture)

There are many build-dependencies loops.

Two kinds of build-dependencies:

  • - host build-deps, to run during the build - target build-deps, e.g. target runtime libraries

lool has An Idea for this: stages for the builds, documented in the debian/control file. Each stage can have different build-depends.

Subject to bitrot as build-dependencies change.

Steve keeps coughing and saying 'multiarch'

cross-build is always necessary to get the toolchain.

Need a tool to plan a build-campaign (other than lamont)

  • - debian and ubuntu have different tools for this - can start with just documentation (?)

*could* extend to be able to build-depend on a package in a particular stage, but probably better to rename packages according to stage... or automagically mangle the version numbers by appending ~stage0 etc

Interface:

  • - stage field on source (list of stages the source can go through)
    • - dont let people name the stages
    - stage field on binary

Do we need to work to prevent accidental installation of ~stageN packages

  • - or just rely on these packages not ending up in a published archive

In early bootstrap phases, building a source package will not necessarily build all of the binary packages that the final build will. ? Not sure what this implies. The existing partition of a source package into binary packages will not already be sufficient for this, do we split the packages in the main archive? Will this be too much overhead. We already split packages for reasons that "don't make sense" like the main/universe. How many packages would need to be split? Probably 30-40.

Some ideas trend in the direction of having stages for the entire archive.

Can't automate the bootstrap of toolchain for sure, automating the bootstrap of build-essential sure to be really hard.

A true, scorched earth, bootstrap has probably only happened 15 times in the last two decades -- how much effort is it sane to put into this?

How to actually specify the stages?

Adapting a package to build in stages will be quite a lot of work.

There is a relationship to cross-builds.

Can use/adapt debcheck to check our progress.

May need to extend launchpad's depwait stuff -- it's currently a bit simplistic and would need to be extended to be aware of stages.

How do we build build-essential? Launchpad's build-dep graph stuff implicitly assumes it's there already! Maybe just write a big script for this.

Is it going to be easier to just build build-essential twice than do a scorched earth rebuild?

Aren't aiming for full-archive scorched archives at first -- but first bit is obviously the hardest.

Three use cases/mode:

  • - really from scratch run binaries, not cross build - cross build - new abi

A prototype would be:

  • - select a set of packages we want to build (e.g. build-essential) - make changes to:
    • - the source packages - dpkg-dev - the build scheduler
      • - this is wannabuild in debian - "soyuz" in launchpad
      - tools that resolve build dependencies:
      • - sbuild - launchpad's 5.5 year old sbuild (or move launchpad to newer sbuild)

ACTIONS

  • lool: spec this more concretely
    • take it to debian prototype
    • dmart: ping steve mcintyre


CategorySpec

Specs/M/ARMAutomatedBootstrap (last edited 2010-12-20 22:59:54 by vorlon)