Ubuntu Wiki

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

• Launchpad Entry: ubuntu-mobile

• Packages affected: moblin-image-creator

Summary

The development process for a mobile device tends to be different then the normal process used for workstation/laptops/servers, where the developer would not normally use the mobile device for their development environment, and where there is need to generate a complete OS image from the developers working environment that is then installed on the target device.

Historically there have been two paths for an embedded Linux developer:

• Use a proprietary tool provided by a commercial embedded Linux vendor
• Use an open tool that builds the complete OS for the specialized embedded device

In the case of the Ubuntu Mobile project, a complete set of built packages is available to the developer, so a lot of the complexities associated with existing build-your-own-os-from-scratch project are just not needed. To bridge the remaining gap between Ubuntu-Mobile development and normal Ubuntu development, this blueprint proposes the use of a new tool called moblin-image-creator.

The moblin-image-creator tool is designed to be extremely flexible with platform specific knowledge isolated to a platform definition. Initial focus is on a new class of devices known as Mobile Internet Devices (MID's), but the design of moblin-image-creator is not MID specific and talk is already in progress to add new platform definitions to build Consumer Electronics stacks such as TV set-top boxes.

There are three fundamental features that moblin-image-creator provides:

• creation of a platform specific build environment
• creation of platform specific target file-systems
• providing user selectable "feature sets" (or fsets) to install bundles of packages that provide some high-level functionality

In addition to this there are many other smaller features to simplify life for the developer, like:

• the choice of a fully functional graphical user interface or a purely command line interface
• wrappers for chrooting into a buildroot or target file-system(i.e. bind mounting important system directories and copying over network configuration files)
• wrappers for opening Xephyr windows for testing target file-systems
• utilities for creating live USB images of target file-systems for easy testing of multiple target file-systems

Release Note

By providing a moblin-image-creator as specified in this blueprint, any Gutsy developer will have the ability to create custom images for a mobile device while at the same time enabling an encapsulated build environment to allow future build reproducibility long after the developers workstation has been updated to future versions of Ubuntu.

Rationale

Without this functionality we force developers to individually create target device filesystems, not only making Ubuntu-Mobile needlessly tedious, but also but also introducing a lot of unnecessary chaos in the community where there will inevitably be many different solutions created to fill the void.

If this functionality is provided upfront in the Ubuntu-Mobile project, then we have a better chance of having the community just fix what's broken in one project then having a large number of mostly working image creation projects.

Use Cases

Developer installs moblin-image-creator from a Gutsy distribution

$ sudo apt-get install moblin-image-creator

Developer uses moblin-image-creator to start a new project

• Developer creates a new project dedicated for working with the targeted device
  • project is configured to work with a mccaslin type of device
  • project is given a name 'myproject' for future reference
  • project is given a description of "My Samsung Q1 Ultra project"
  • project creates a complete buildroot in /usr/src/myproject

$ sudo image-creator -c create-project                  \
                      --platform-name mccaslin            \
                      --project-name "myproject"          \
                      --project-path "/usr/src/myproject" \
                      --project-description "My Samsung Q1 Ultra project" 

Developer uses moblin-image-creator to create a new target filesystem

• Developer creates a new initial target device filesystem
  • since a project can create multiple target filesystems, the target is named 'target1'
  • the new target filesystem is rooted at /usr/src/myproject/targets/target1/fs

$ sudo image-creator -c create-target          \
                      --project-name "myproject" \
                      --target-name "target1"

Developer installs the Ubuntu-Mobile default set of packages for a full UI

• Developer installs the "full-mobile-stack" functional set (fset)
  • The "full-mobile-stack" is a meta fset that depends on the Core, Infrastructure, GNOME-Mobile, Mobile-Applications, and Power-Management fsets
  • after installing this fset, the target filesystem has enough functionality such that once installed on the target device, the device can boot to the ubuntu-mobile installation and automatically startup into the hildon desktop with the standard set of ubuntu-mobile applications

$ sudo image-creator -c install-fset         \
                      --project-name myproject \
                      --target-name target1    \
                      --fset-name "full-mobile-stack"

• Developer tweaks the target filesystem as desired
  • The developer has the choice of building new mobile software from the project buildroot (which comes with the basic ubuntu-mobile development packages installed), or building from their Gutsy host.
  • The software will then need to be installed in the target filesystem. If building from the buildroot, then the target filesystem is located at /targets/target1/fs, and if building from a Gutsy host, then the target filesystem is located at /usr/src/myproject/targets/target1/fs

Developer builds test the target filesystem on host system

• Developer installs the Xephyr server in their workstation and starts a new Xephyr window

$ sudo apt-get install xserver-xephyr
$ Xephyr :2 -host-cursor -screen 800x480x16 -dpi 96 -ac &

• Developer uses moblin-image-creator to chroot inside the target filesystem
  • moblin-image-creator will handle all the mundane details beyond just calling chroot
    • bind mounting /proc, /sys/, and /tmp
    • copying over host network configuration files
    • starting target daemons like dbus
    • creating a fresh environment such that applications are not broke with host specific settings

$ sudo image-creator -c chroot-target        \
                      --project-name myproject \
                      --target-name target1
Password: XXXXX
# 

• Developer launches the hildon-desktop

# start-hildon-desktop &

Developer Creates an Installation Image

• moblin-image-creator provides several mechanisms for installing the new target filesystem on the device, including:
  • USB Install (boot a USB key that installs the target filesystem on the device)
  • Live USB (boot a working live image off a USB key)
  • Live RW USB (boot a working live image off a USB key that persist changes to the key)
  • ISO Install (user boots off a USB CD that installs the target filesystem on the device)
  • Live ISO (boot a working live image off a CD)
• For this usage case, the developer creates an "Install USB" image and then writes the image
  • to a USB key that shows up as /dev/sdb on the developers workstation

$ sudo image-creator -c create-install-usb   \
                      --project-name myproject \
                      --target-name target1    \
                      --image-name live-usb.img
$ sudo dd if=/usr/src/myproject/targets/target1/image/live-usb.img of=/dev/sdb

Developer Installs the target filesystem on the device

• Developer plugs in the USB key to the device and then boots device and enters the BIOS configuration and configures the boot options to boot off a USB key
• The device boots, automatically installs the target filesystem, and then shutdown
• The developer unplugs the USB key and boots the device to see the running ubuntu-mobile stack

Build system generates an image for a given device

• the build system uses the moblin-image-creator command line capabilities to create target images

# Inside the build script....
# $PLATFORM is the name of the platform
# $BUILDROOT is the build working directory
# $DEST is the directory to place the target image

# Delete any existing build project
image-creator -c delete-project --project-name build 2> /dev/null || true

# Kick off a clean project
image-creator -c create-project        \
               --platform-name $PLATFORM \
               --project-name build      \
               --project-path $BUILDROOT \
               --project-description "Build system" 

# Install a new target in the project
image-creator -c create-install-usb    \
               --project-name build      \
               --target-name buildtarget \
               --image-name install.img

# Install the standard mobile stack in the target
image-creator -c install-fset          \
               --project-name build      \
               --target-name buildtarget \
               --fset-name "full-mobile-stack"

# generate the an install USB image
image-creator -c create-install-usb    \
               --project-name build      \
               --target-name buildtarget \
               --image-name install.img

mv $BUILDROOT/targets/buildtarget/image/live-usb.img $DEST

Assumptions

• moblin-image-creator users are developers with at least a basic understanding of how Linux systems work
• the user's workstation has internet access, and if run from inside a private network, then the system wide proxy is correctly configured on the workstation

Design

The project builder is composed of a set of python libraries that abstract the fundamental operations into classes, and a top-level python application that will either kick off a Graphical User Interface (GUI) if no command line arguments are given, or process the command line arguments to perform the requested task.

SDK Class

The SDK class is a toplevel class that provides the ability to:

• create a new project
• delete an existing project
• provide the path to the moblin-image-creator shared data
• provide a list of all available platform objects
• provide a list of all available project objects

Platform Class

The platform class abstracts the configuration data defined by a platform by providing:

• platform meta data like name and description
• a list of packages that should be installed in the project buildroot
• a list of apt configuration files that the project buildroot should utilize

Filesystem Class

The filesystem class abstracts any given filesystem, providing:

• ability to install a list of packages
• ability to setup the filesystem such that applications running inside a chroot of the filesystem will function as expected. For example:
  • bind mounting important host system directories like /proc, /sys, and /tmp
  • copying over import configuration file like the network resolv.conf file
  • establishing a fresh environment
• ability to take a given command, and execute it inside a chroot of the associated filesystem

Project Class (is a Filesystem)

The project class abstracts all aspects of a moblin-image-creator project, providing:

• ability to install packages in a project buildroot
• ability to update the buildroot apt database
• ability to create a new target
• ability to delete an existing target
• ability to create a new install image from an existing target
• access to the platform object associated with the project
• access to project meta data

Target Class (is a Filesystem)

The target class abstracts all aspects of a target filesystem, providing:

• ability to install functional sets (fsets) in a the associated target filesystem
• ability to list currently installed fsets
• ability to update the associated target filesystem

FSet Class

A functional set (fset) is the mechanism for exposing groups of functionality to the user, where instead of having the user pick individual packages, the user selects an fset that resolves to a group of packages, some optional debug packages, and dependencies on other fsets.

The fset class abstracts this by providing:

• ability to read fset configuration files that add new fsets
• provide a list of fset dictionaries, where an fset dictionary contains:
  • DESC (a description of the fset)
  • PKGS (a list of required packages)
  • DEBUG_PKGS (a list of optional debug packages)
  • DEPS (a list of dependent fsets)

Implementation

The moblin-image-creator tool can be used as either a command line tool, or as a GUI if no command line arguments are provided. For a list of available command line arguments, use the --help argument:

$ image-creator --help
Usage: image-creator [options]

Options:
  -c CMD, --command=CMD
                        Where CMD is one of: chroot-project, chroot-target,
                        create-install-iso, create-install-usb, create-live-
                        iso, create-live-usb, create-project, create-target,
                        delete-project, delete-target, install-fset, list-
                        fsets, list-platforms, list-projects, list-targets,
                        update-project, or update-target
  --platform-name=PLATFORM_NAME
                        Platform name
  --project-name=PROJECT_NAME
                        Project name
  --project-description=PROJECT_DESC
                        Project description
  --project-path=PROJECT_PATH
                        Project path
  -t TARGET_NAME, --target-name=TARGET_NAME
                        Target name
  --fset-name=FSET_NAME
                        Feature set identifier
  --image-name=IMAGE_NAME
                        Name to use for target image file
  -q, --quiet           don't print status messages to stdout
  -d, --enable-debug    Enable additional debug package while installing fsets
  -h, --help            show this help message and exit


Examples:
<Adding a new project>
    image-creator --command=create-project \
                    --platform-name='mccaslin' \
                    --project-name='MyProject' \
                    --project-desc='Example project' \
                    --project-path=/usr/src/projects/myproject

<Delete a project>
    image-creator --command=delete-project \
                    --project-name='MyOtherProject'

<Adding a new target to an existing project>
    image-creator --command=create-target \
                    --project-name='MyProject' \
                    --target-name='MyTarget'

<Delete a target>
    image-creator --command=delete-target \
                    --project-name='MyProject' \
                    --target-name='MyOtherTarget'

<installing an fset into a given target>
    image-creator --command=install-fset \
                    --platform-name='mccaslin' \
                    --project-name='MyProject' \
                    --target-name='MyTarget' \
                    --fset='Core' \

<change into a given project buildroot filesystem>
    image-creator --command=chroot-project \
                    --project-name='MyProject' \

<change into a given projects target filesystem>
    image-creator --command=chroot-target \
                    --project-name='MyProject' \
                    --target-name='MyTarget' \

<updating a given target inside a project>
    image-creator --command=update-target \
                    --project-name='MyProject' \
                    --target-name='MyTarget' \

<updating a given project>
    image-creator --command=update-project \
                    --project-name='MyProject'

GUI Screen Shots

attachment:project-builder-window.png

attachment:project-builder-full-desktop.png

Shared Directory Layout

The /usr/share/pdk directory is used as a shared directory for moblin-image-creator. It contains:

• lib/ directory containing the moblin-image-creator library files
• pixmaps/ directory containing images used by the GUI
• locale/ directory containing language data

• help/ directory containing DocBook XML files for the GNOME yelp utility

• platforms/ directory contain platform definition data
• moblin-image-creator.glade file used by the GUI
• repo/ directory containing an apt repository for any platform specific configuration
  • packages that for one reason or another are not already in the Ubuntu repository

Buildroot and Target Rootstraps

In order to both speed up the task of creating project buildroots and initial target filesystems, the moblin-image-creator build will use debootstrap to construct an initial filesystem used for the buildroot (that uses the buildd variant) and another initial filesystem that is used for the initial target filesystem.

The process of creating a new project entails unarchiving the platform specific buildroot rootstrap, chrooting inside to call 'apt-get update', and then installing the list of additional packages in the platform's buildroot.packages configuration file.

The process of creating a new target entails unarchiving the platform specific target rootstrap and then chrooting inside to call 'apt-get update'.

UI Changes

Since this is a new project, there are no UI changes.

Code Changes

Since this is a new project, there are no code changes.

Migration

Since this is a new project, there is no such thing as data migration.

Test/Demo Plan

The moblin-image-creator source contains a set of unit test that utilize the python 'unittest' module that is already available in Ubuntu. A subset of the unit test (composing test that are considered to prove basic acceptance criteria) are automatically executed in the top level makefiles 'install' target. Since the debian rule file uses this target, then debian package builds will fail if and of the basic acceptance test fail.

Developers can also run the basic test by executing the 'runbasictest' toplevel makefile target, or execute the full portfolio of unit test by executing the 'runalltest' target.

$ sudo make runbasictest
... basic acceptance test will run ...
$ sudo make runalltest
... all unit test will be run ...

Required Unit Test

Every time a new class is added to the moblin-image-creator python libraries, at least one basic acceptance test needs to be created, and ideally a series of further unit test should be added.

Basic Acceptance Test

A basic acceptance unit test must:

• be relatively quick (where creating a full target filesystem is most definitely not quick enough)
• provide rudimentary level of testing

At the very least, just importing the file containing the new class and instantiating the new class will expose syntax errors that will tank the python interpreter at run time.

Outstanding Issues

Since the moblin-image-creator needs to create rootstrap images in the debian package build process, then it needs to have root privileges. A default pbuilder configuration will attempt to build the package as a non-root user (for a lot of good reasons.) Unless the Ubuntu build is configured to run as root, then this packages debian rules will need to be reworked.

CategorySpec