Ubuntu Wiki

• Launchpad Entry: arm-m-image-building-tool

• Created:

• Contributors: MichaelHudson, JamesWestby

• Packages affected: live-helper

Summary

We will provide and document an easy-to-use tool to build system images.

This is a much a matter of providing examples, writing documentation, establishing conventions and best practice and social engineering as it is software development task. There are variety of tools for this already, but we will pick one and polish it until it is suitable to become a standard.

Rationale

Being able to easily build images is required to make developing on ARM easy, as it it sometimes the only way to test a fix. In addition, because ARM has so many flavors that require different images it means we need something scalable and extensible. The existing landscape of tool-building images consists of a wide variety of scripts of various levels of generality and quality. The aim is to pick a chosen tool to put in the hands of developers, and then improve it such that images can be created with minimum developer effort.

User stories

Developer John has made some changes to the kernel and needs to build an image to flash onto a SD card so he can test boot times on his prototype hardware. He makes a simple change to a configuration file, runs a shell command and use dd to blat the image onto the SD card.

Assumptions

• We will have the ability to run code as root during the image build process (although see this spec)

• We assume the image will be built in its native environment, either on ARM hardware or using emulation.

Design

We will standardise on live-helper.

The two issues identified that are most pressing for us to improve in the tool are support for ARM bootloaders, and improved configuration.

Bootloaders

live-helper does not yet support ARM boot-loaders, so we need to extend it to be able to create images using these bootloaders. It already has support for multiple bootloaders, so it will be a case of adding some more options. There will be some scope for automatic detection (is the architecture "armel", but user selection should be allowed too). We aim to standardise on u-boot, so that will be the bootloader that we default to on armel.

=== Configuration ====

The current live-helper configuration format is rather verbose, with lots of files and directories being created to just have the defaults. This makes it harder for developers to see what is specific to their project, and what is just defaults.

Therefore we would like to be able to use a simplified configuration format that can just be a minimal configuration for each project if desired.

While this makes it easier to comprehend, it does lose the self-documenting of other options that are currently defaults. Therefore we will also ensure that these are documented elsewhere, not just in the code. At first this may just be providing access to a full default configuration for comparison, but there may be better ways to present the information.

The first step will be to ensure that live-helper can work from a minimised configuration file.

After that the creation of a new project should be changed to output a minimal file, rather than include all the defaults.

Once we have that we can discuss with developers whether they want an alternative format. Currently the format is very low-level, requiring you to look at the details. It's possible that a higher-level configuration interface would be desired, breaking images in to classes or similar, such that you can pick a class and get several options set correctly for that class, but still with the ability to tweak specific options.

Implementation

Bootloaders

There needs to be a small live-helper script created, lh_binary_uboot, along the same lines as the other bootloader scripts (lh_binary_grub etc.) This will install uboot as the bootloader for the image.

Then live-helper needs to be modified to default to using the script on armel if the choice of bootloader isn't overridden.

Configuration

First we check that live-helper can work from a minimal configuration. This involves testing and checking the code to see if it assumes that variables are set, rather than checking whether they are empty before using them. Unfortunately this is not easy to do programatically, but there isn't too much to inspect by eye.

The the code should be changed

Migration

We're only targeting the use of the new tool at new projects.

Test/Demo Plan

Bootloaders

There will be tests done to ensure that uboot can be installed by default, by explicit selection, and can be overriden to install something else on armel.

Also where there are options for the way that it is installed each will be tested to ensure that the choice is passed through from the configuration file to the right options on disk.

Configuration

Unresolved Issues

UDS discussion

There seemed to be consensus that the tool would be built on live-helper, which despite being awkward to configure and not yet supporting ARM bootloaders, is maintained and flexible, has decent docs and a sane architecture, and generally seemed the most promising.

Other bases considered were:

• rootstock (only builds filesystems, not images)
• livecd-rootfs (requires two phases, basically unconfigurable)
• vmbuilder (not so good at images requiring specific partition layout)

All tools are built on debootstrap. It might make sense to base them on multistrap instead:

• - it uses apt+dpkg instead of reimplementing parts - can use multiple archives - qemu-debootstrap exists, and should perhaps be ported. - live-helper has plans to use multistrap. - it does mean you have problems with pre-depends and dpkg-diverts - multiple archives is nice, but there are other ways to do that, and
  • live-helper makes that easy.

live-helper v2 is in development and although only alpha probably makes sense as the base of our system.

live-helper configuration is too complex to get started. Abstractions might be good.

• - perhaps think dh v7, where the tool does the sensible thing by default, and you can just add overrides where you have unusual needs. - Many files/folders by default, providing a way to have a single configuration file that only specifies the necessary things would make it easier to grasp what the project requires.
  • - it probably makes sense to try to get these changes done upstream

Reproducing previous builds given the same inputs is desirable but tricky.

• - OEM finds that some companies want to be able to do it. - Want to be able to make a single change. - Snap-shots (like snapshot.debian.org) could be an option but requires disk space.

Actions

This is from the gobby notes, but they are (a) a bit vague and (b) not assigned to anyone, so I don't know how to weave them into the spec.

• virtual ppa infrastructure needed for ARM because image building requires building parts as root
  • virtualized QEMU is possible but slow.
• add support for ARM bootloaders to live-helper
• write a proposal for image building service (DB, web ui, etc.)
• Look into livehelper 2
• Investigate easier config for live-helper
• Simplification of live-helper usage (configuration) for common targets.

CategorySpec