Debian/Ubuntu Boot Performance Sprint


Canonical's Millbank Office, London, UK


  • Monday & Tuesday, June 22/23, 2009

  • 9am - 6pm


Investigate creating a joint Ubuntu-Debian team to look into how both distributions can work together to improve boot performance.


Topics of Discussion

  • Current boot speed of Ubuntu and Debian
    • Comparison of bootchart data
    • Analyze differences
  • Future plans for Ubuntu and Debian
    • sysvinit
    • upstart
  • Ways Debian and Ubuntu can leverage each others progress
  • Next Steps?
    • DebConf

    • How to continue communication, i.e. mailing list, wiki page, etc?


Considerations for Transitioning Debian to Use upstart

  • Preconditions for /sbin/init as upstart:
    • /etc/inittab backwards-compatibility support
      • should reuse the existing inittab conversion script to spit out jobs to a tmp dir that upstart reads from
      • inotify watch /etc/inittab, generate event when it changes and hook up an upstart job to run the conversion script
    • sysv-rc support for upstart jobs (/lib/init/upstart-job)
    • dh_installupstart
    • update-rc.d, invoke-rc.d implementation that can DTRT with upstart jobs
  • a soft migration in Debian will be more easily accepted
  • could have upstart read /etc/init.d/* as a directory of conf files in a different "format" and parse the LSB headers
    • LSB expressed interest in this approach
  • could have a "translator" app to allow sysvinit to execute /etc/init/*.conf files, see upstart-job for sysvinit.
  • What we do with the non-Linux port?
    • Get upstart working with BSD implement upstart job compat with sysvinit
    • upstart/sysvinit bi-directional dependency handling

Transition from sysvinit to upstart

sysvinit-upstart relationship drawing

  • /lib/init/upstart-job is either:
    • (sysvinit installed) a helper utility built from upstart source that provides a wrapper for running the named upstart service as if it were an init script
    • (upstart installed) a helper utility that will warn the user about using /etc/init.d/$JOB and run 'service $JOB start' instead
  • upstart-job needs an option to spit out the LSB headers to feed them to things like insserv
  • insserv needs to check, if /etc/init.d/foo is a symlink to /lib/init/upstart-job and in that case call the init script with the right option to get LSB header and not directly parse the file (proposed option name: '/lib/init/upstart-job <service basename> lsb-header')

  • dh_installupstart to autogenerate an init script symlink shipped in the package
    • Add dependency on upstart-job to misc:Depends
    • Generate symlink /etc/init.d/foo -> /lib/init/upstart-job

  • dh_installupstart also has to provide the following, see also

    • packages include debian/*.upstart, these are automatically installed to the correct location
    • a previous /etc/event.d file is automatically removed or backed up
    • a previous /etc/init.d script is automatically removed or backed up
    • maintainer script commands for placing the job in manual mode across an upgrade are automatically provided
    • need to run the dh_installupstart design past joeyh early
  • upstart-job is provided by a package that provides virtual package (prob. upstart-job), this can be upstart or sysv-compat-upstart depending which version is required
  • upstart-job must be buildable on all architectures and should be part of the upstart source package. For architectures where upstart currently fails to builds (like kfreebsd/hurd), it should be possible to disable compilation.
  • /lib/init/upstart-job should have no dependencies on anything in /usr, should be possible to be implemented in shell, but preferably implemented in C for speed reasons.
  • packages shipping upstart job files must depend on this virtual package (handled by the debhelper script)

/etc/inittab Support

  • runs between rcS and rc2
  • based on existing migration script
  • copies from /etc/event.d (... /etc/init?) to /lib/init/rw/event.d and rewrites
  • adds new files to /lib/init/rw/event.d
  • upstart reads the latter directory via inotify
  • need signal that instructs upstart to re-run that script (init q)

Reviewed of Debian Unstable Bootchart

  • debian-sid-makefile-concur-2009-06-22.png

    • hwclock &

    • why is modprobe slow in the initramfs?
    • x-session starting about 4 secs after Xorg (gdm bug)
    • need to install a newer version of readahead

      • current readahead has problems which cause boot failure under certain circumstances
      • using sreadahead would provide an improvement on SSDs
    • loadcpufreq possibly not needed with some kernel tuning

service Command

  • initial implementation available in Ubuntu, implements the traditional start/stop/status commands
  • the service script should be a common implementation in sysvinit-utils, the common Essential: yes package
  • one backend implementation for each rc system (sysv-rc, file-rc, insserv, and upstart), via update-rc.d
    • spec out the interface first, then migrate functionality into the rc system packages
    • documentation for update-rc.d (policy, manpages) should be explicit that these interfaces are internal to service, and are not meant for use by maintainer scripts (... nor users)


  • enable|disable feature - if provided at update-rc.d layer we need to ensure sysvinit-utils depend on versions of packages which provide update-rc.d with enable|disable feature (Provides?)
  • uncertainty and doubt whether update-rc.d is correct layer to contain the disable|enable implementations
    • can reuse existing code within update-rc.d
    • update-rc.d should not become more appealing to users - it is not intended for them to use. Adding this functionality does just that however ...
    • need to define new interfaces in policy?

service Backend / update-rc.d Extensions Requirements

  1. Enable/disable service (user-facing)

    • [2345]/Sxx <-> [2345]/K(100-xx)

  2. Remove link (usually 0/6, but not always) (maintainer script)

    • R/[SK]xx -> null

  3. Add link, e.g. due to bugs such as copying RH runlevel setup (maintainer script)

    • null -> R/[SK]xx

  4. Move link in sequence (maintainer script)

    • R/[SK]xx -> R/[SK]yy (atomic)

  5. Move link between runlevels (maintainer script)

    • X/Sxx -> Y/Syy (atomic)

update-rc.d Syntax

Explored several possibilities for update-rc.d syntax to edit links: general idea is that you have one correct current state, but also some previous valid states which should be considered as package-generated rather than user-configured

update-rc.d service start 80 S . was start 99 2 3 4 5 .

if dpkg --compare-versions "$2" lt blah; then
    PREV='was start 99 2 3 4 5 .'
update-rc.d start 80 S . $PREV

update-rc.d service start 30 2 3 4 5 .
if dpkg --compare-versions "$2" lt blah; then
    update-rc.d service replace start 20 2 3 4 5 . 30 2 3 4 5 .
  • Difficult case is when you have multiple logically-distinct changes; e.g. pcmciautils wants to remove 0 and 6 links (teardown), and, independently, move rcS.d/S40 to rcS.d/S13 (Ubuntu boot ordering)
  • Petter suggests a replace command which can express a change rather than a previous state

  • Useful to have dh_installinit deal with version check generation, so that no maintainer script code needs to be written by hand

Actions Required by service Backend

  • enable (service should run automatically from now on)
  • disable (service should not run automatically from now on)
  • query (output representation of current state of service)
  • start (start service immediately, regardless of policy)
  • stop (stop service immediately, regardless of policy)
  • status (enquire whether service is currently running, as in /etc/init.d/service status)

  • list-services (list all service names)

    - /usr/lib/service/backend as interface name
    - Petter wants to avoid races between multiple diversions (possible solution: symlink, though this means that everything ends up in update-rc.d which would encourage people to use it; may be partially soluble with good documentation)

Debian sysvinit ToDo List

  • revert mtab -> /proc/mounts symlink patch => requires further discussion. It seems to break things currently. (Unstable fixed 2009-06-23)

  • fsck should run even if on_ac_power #526398 (Unstable fixed 2009-06-23)
  • devpts in -> non-udev case? need to check

  • move startpar to sysvinit-utils (Unstable fixed 2009-06-23)
  • contact openSuSE boot people to find out if they have plans to move away from sysvinit[?]
  • cleanup debian/rules, debian/patches/* anyone?
    • move from dpatch -> quilt, conform with new package format standards (eg. do not patch ./debian/* !)

    • use a bit of debhelper in debian/rules

Bootchart Package

  • Scott has written a new bootchart which is more featureful, we should consider adopting it
  • currently packaged bootchart is dead upstream
  • currently packaged bootchart does not collect information from kernel/initramfs phase
  • split source packages ? ubuntu have bootchart (the collector), bootchart-java (the java chart 'renderer' that is currently in Debian)
  • contact current bootchart Debian maintainer and let him know about Scott's new work [kel]
  • need to reliably test and record boot performance on Debian [kel]
    • Canonical loaned Kel a Dell Mini 9 for this work Smile :)

Dash as /bin/sh

  • communicate to users of Debian why using dash as /bin/sh is a good thing. Ubuntu stuff =>

  • raphael has done much work in Debian to make it possible for squeeze, contact him about his current opinion in regards to making it a default

insserv in Ubuntu to Validate LSB-header-declared Dependencies?

<cjwatson@sarantium ~>$ /usr/share/insserv/check-initd-order
error: Unable to read /etc/rc2.d/S81update-motd at /usr/share/insserv/check-initd-order line 373.
LSB header missing in /etc/rcS.d/S07linux-restricted-modules-common
Incorrect order checkroot@20 > /etc/rcS.d/S15module-init-tools15
LSB header missing in /etc/rc2.d/S10powernowd.early
Incorrect order avahi@50 > /etc/rc2.d/S28libvirt-bin28
LSB header missing in /etc/rc2.d/S30gdm
error: Unable to read /etc/rc2.d/S81update-motd at /usr/share/insserv/check-initd-order line 373.
LSB header missing in /etc/rc2.d/S81update-motd
LSB header missing in /etc/rc2.d/S99acpi-support
<cjwatson@sarantium ~>$ /usr/share/insserv/check-initd-order -k
LSB header missing in /etc/rc6.d/K01gdm
  • checkroot/module-init-tools: looks like a bug in module-init-tools saying Should-Start: checkroot
  • avahi/libvirt-bin: libvirt-bin needs avahi, started at wrong point?

Dependency graph: /usr/share/insserv/check-initd-order -g | dot -Tpdf > init.pdf

LSB headers but no dependencies: kvm (Ubuntu #391121), apparmor

Communication Plan

  • Send a summary out to ubuntu-devel[robbie] and debian-devel about this meeting and the outcomes
  • Provide the reasons WHY we care about boot performance [scott]
  • Highlight the reasons (not just bootspeed) why switching to upstart is a *good* thing (e.g. reliability, coping with an async kernel)
  • Continue discussions on:

FoundationsTeam/BootPerformance/DebianUbuntuSprint (last edited 2009-07-08 15:30:01 by e180076113)