What

Debian/Ubuntu Boot Performance Sprint

Where

Canonical's Millbank Office, London, UK

When

• Monday & Tuesday, June 22/23, 2009

• 9am - 6pm

Why

Investigate creating a joint Ubuntu-Debian team to look into how both distributions can work together to improve boot performance.

Who

• ColinWatson

• KelModderman

• LukClaes

• MichaelBiebl

• PetterReinholdtsen

• RobbieWilliamson

• ScottJamesRemnant

• SteveLangasek

Topics of Discussion

• Current boot speed of Ubuntu and Debian
  • Comparison of bootchart data
  • Analyze differences
• Future plans for Ubuntu and Debian
  • sysvinit
  • upstart
• Ways Debian and Ubuntu can leverage each others progress
• Next Steps?

  • DebConf

  • How to continue communication, i.e. mailing list, wiki page, etc?

Notes

Considerations for Transitioning Debian to Use upstart

• Preconditions for /sbin/init as upstart:
  • /etc/inittab backwards-compatibility support
    • should reuse the existing inittab conversion script to spit out jobs to a tmp dir that upstart reads from
    • inotify watch /etc/inittab, generate event when it changes and hook up an upstart job to run the conversion script
  • sysv-rc support for upstart jobs (/lib/init/upstart-job)
  • dh_installupstart
  • update-rc.d, invoke-rc.d implementation that can DTRT with upstart jobs
• a soft migration in Debian will be more easily accepted
• could have upstart read /etc/init.d/* as a directory of conf files in a different "format" and parse the LSB headers
  • LSB expressed interest in this approach
• could have a "translator" app to allow sysvinit to execute /etc/init/*.conf files, see upstart-job for sysvinit.
• What we do with the non-Linux port?
  • Get upstart working with BSD implement upstart job compat with sysvinit
  • upstart/sysvinit bi-directional dependency handling

Transition from sysvinit to upstart

sysvinit-upstart relationship drawing

• /lib/init/upstart-job is either:
  • (sysvinit installed) a helper utility built from upstart source that provides a wrapper for running the named upstart service as if it were an init script
  • (upstart installed) a helper utility that will warn the user about using /etc/init.d/$JOB and run 'service $JOB start' instead
• upstart-job needs an option to spit out the LSB headers to feed them to things like insserv

• insserv needs to check, if /etc/init.d/foo is a symlink to /lib/init/upstart-job and in that case call the init script with the right option to get LSB header and not directly parse the file (proposed option name: '/lib/init/upstart-job <service basename> lsb-header')

• dh_installupstart to autogenerate an init script symlink shipped in the package
  • Add dependency on upstart-job to misc:Depends

  • Generate symlink /etc/init.d/foo -> /lib/init/upstart-job

• dh_installupstart also has to provide the following, see also https://wiki.ubuntu.com/FoundationsTeam/Specs/KarmicUpstartPackagingPolicy

  • packages include debian/*.upstart, these are automatically installed to the correct location
  • a previous /etc/event.d file is automatically removed or backed up
  • a previous /etc/init.d script is automatically removed or backed up
  • maintainer script commands for placing the job in manual mode across an upgrade are automatically provided
  • need to run the dh_installupstart design past joeyh early
• upstart-job is provided by a package that provides virtual package (prob. upstart-job), this can be upstart or sysv-compat-upstart depending which version is required
• upstart-job must be buildable on all architectures and should be part of the upstart source package. For architectures where upstart currently fails to builds (like kfreebsd/hurd), it should be possible to disable compilation.
• /lib/init/upstart-job should have no dependencies on anything in /usr, should be possible to be implemented in shell, but preferably implemented in C for speed reasons.
• packages shipping upstart job files must depend on this virtual package (handled by the debhelper script)

/etc/inittab Support

• runs between rcS and rc2
• based on existing migration script
• copies from /etc/event.d (... /etc/init?) to /lib/init/rw/event.d and rewrites
• adds new files to /lib/init/rw/event.d
• upstart reads the latter directory via inotify
• need signal that instructs upstart to re-run that script (init q)

Reviewed of Debian Unstable Bootchart

• debian-sid-makefile-concur-2009-06-22.png

  • hwclock & hwclock.sh....why?

  • why is modprobe slow in the initramfs?
  • x-session starting about 4 secs after Xorg (gdm bug)

  • need to install a newer version of readahead

    • current readahead has problems which cause boot failure under certain circumstances
    • using sreadahead would provide an improvement on SSDs
  • loadcpufreq possibly not needed with some kernel tuning

service Command

• initial implementation available in Ubuntu, implements the traditional start/stop/status commands

  • bug filed in Debian: #534300

• the service script should be a common implementation in sysvinit-utils, the common Essential: yes package
• one backend implementation for each rc system (sysv-rc, file-rc, insserv, and upstart), via update-rc.d
  • spec out the interface first, then migrate functionality into the rc system packages

  • documentation for update-rc.d (policy, manpages) should be explicit that these interfaces are internal to service, and are not meant for use by maintainer scripts (... nor users)

• http://lists.debian.org/debian-devel/2009/04/msg00017.html

• enable|disable feature - if provided at update-rc.d layer we need to ensure sysvinit-utils depend on versions of packages which provide update-rc.d with enable|disable feature (Provides?)
• uncertainty and doubt whether update-rc.d is correct layer to contain the disable|enable implementations
  • can reuse existing code within update-rc.d
  • update-rc.d should not become more appealing to users - it is not intended for them to use. Adding this functionality does just that however ...
  • need to define new interfaces in policy?

service Backend / update-rc.d Extensions Requirements

1. Enable/disable service (user-facing)
   

   • [2345]/Sxx <-> [2345]/K(100-xx)

2. Remove link (usually 0/6, but not always) (maintainer script)
   

   • R/[SK]xx -> null

3. Add link, e.g. due to bugs such as copying RH runlevel setup (maintainer script)
   

   • null -> R/[SK]xx

4. Move link in sequence (maintainer script)
   

   • R/[SK]xx -> R/[SK]yy (atomic)

5. Move link between runlevels (maintainer script)
   

   • X/Sxx -> Y/Syy (atomic)

update-rc.d Syntax

Explored several possibilities for update-rc.d syntax to edit links: general idea is that you have one correct current state, but also some previous valid states which should be considered as package-generated rather than user-configured

update-rc.d service start 80 S . was start 99 2 3 4 5 .

if dpkg --compare-versions "$2" lt blah; then
    PREV='was start 99 2 3 4 5 .'
fi
update-rc.d start 80 S . $PREV

update-rc.d service start 30 2 3 4 5 .
if dpkg --compare-versions "$2" lt blah; then
    update-rc.d service replace start 20 2 3 4 5 . 30 2 3 4 5 .
fi

• Difficult case is when you have multiple logically-distinct changes; e.g. pcmciautils wants to remove 0 and 6 links (teardown), and, independently, move rcS.d/S40 to rcS.d/S13 (Ubuntu boot ordering)

• Petter suggests a replace command which can express a change rather than a previous state

• Useful to have dh_installinit deal with version check generation, so that no maintainer script code needs to be written by hand

Actions Required by service Backend

• enable (service should run automatically from now on)
• disable (service should not run automatically from now on)
• query (output representation of current state of service)
• start (start service immediately, regardless of policy)
• stop (stop service immediately, regardless of policy)

• status (enquire whether service is currently running, as in /etc/init.d/service status)

• list-services (list all service names)

  - /usr/lib/service/backend as interface name
  - Petter wants to avoid races between multiple diversions (possible solution: symlink, though this means that everything ends up in update-rc.d which would encourage people to use it; may be partially soluble with good documentation)
  

Debian sysvinit ToDo List

• revert mtab -> /proc/mounts symlink patch => requires further discussion. It seems to break things currently. (Unstable fixed 2009-06-23)

• fsck should run even if on_ac_power #526398 (Unstable fixed 2009-06-23)

• devpts in mountvirtfs.sh -> non-udev case? need to check

• move startpar to sysvinit-utils (Unstable fixed 2009-06-23)
• contact openSuSE boot people to find out if they have plans to move away from sysvinit[?]
• cleanup debian/rules, debian/patches/* anyone?

  • move from dpatch -> quilt, conform with new package format standards (eg. do not patch ./debian/* !)

  • use a bit of debhelper in debian/rules

Bootchart Package

• Scott has written a new bootchart which is more featureful, we should consider adopting it
• currently packaged bootchart is dead upstream
• currently packaged bootchart does not collect information from kernel/initramfs phase
• split source packages ? ubuntu have bootchart (the collector), bootchart-java (the java chart 'renderer' that is currently in Debian)
• contact current bootchart Debian maintainer and let him know about Scott's new work [kel]
• need to reliably test and record boot performance on Debian [kel]

  • Canonical loaned Kel a Dell Mini 9 for this work

Dash as /bin/sh

• communicate to users of Debian why using dash as /bin/sh is a good thing. Ubuntu stuff => https://wiki.ubuntu.com/DashAsBinSh

• raphael has done much work in Debian to make it possible for squeeze, contact him about his current opinion in regards to making it a default

insserv in Ubuntu to Validate LSB-header-declared Dependencies?

<cjwatson@sarantium ~>$ /usr/share/insserv/check-initd-order
error: Unable to read /etc/rc2.d/S81update-motd at /usr/share/insserv/check-initd-order line 373.
LSB header missing in /etc/rcS.d/S07linux-restricted-modules-common
Incorrect order checkroot@20 > /etc/rcS.d/S15module-init-tools15
LSB header missing in /etc/rc2.d/S10powernowd.early
Incorrect order avahi@50 > /etc/rc2.d/S28libvirt-bin28
LSB header missing in /etc/rc2.d/S30gdm
error: Unable to read /etc/rc2.d/S81update-motd at /usr/share/insserv/check-initd-order line 373.
LSB header missing in /etc/rc2.d/S81update-motd
LSB header missing in /etc/rc2.d/S99acpi-support
<cjwatson@sarantium ~>$ /usr/share/insserv/check-initd-order -k
LSB header missing in /etc/rc6.d/K01gdm

• checkroot/module-init-tools: looks like a bug in module-init-tools saying Should-Start: checkroot
• avahi/libvirt-bin: libvirt-bin needs avahi, started at wrong point?

Dependency graph: /usr/share/insserv/check-initd-order -g | dot -Tpdf > init.pdf

LSB headers but no dependencies: kvm (Ubuntu #391121), apparmor

Communication Plan

• Send a summary out to ubuntu-devel[robbie] and debian-devel about this meeting and the outcomes
• Provide the reasons WHY we care about boot performance [scott]
• Highlight the reasons (not just bootspeed) why switching to upstart is a *good* thing (e.g. reliability, coping with an async kernel)
• Continue discussions on:
  • IRC: #pkg-sysvinit@OFTC

  • Mailing List: initscripts-ng-devel@lists.alioth.debian.org