Reducing Boot Time of UME Image

Summary of Problem

Looking at the bootchart for a UME image, you can see that modprobe runs for a significant portion of the boot process. Around second 16, there are even 3 modprobe instances running concurrently. udev has a rule that calls modprobe anytime it receives a kernel uevent for a device with an associated module alias. This rule is the last rule in /etc/udev/rules.d/90-modprobe.rules. On cold boot, /etc/rcS.d/S10udev calls udevtrigger to trigger a kernel device uevent for all devices. This results in modprobe being called for all devices with a module alias--even if the device's module is not needed to boot. The snd_hda_intel is one such module and is particularly troublesome because it causes all of the following sound-related modules to be loaded:

• soundcore
• snd
• snd_seq_device
• snd_timer
• snd_hwdep
• snd_page_alloc
• snd_seq
• snd_seq_midi_event
• snd_pcm
• snd_rawmidi
• snd_mixer_oss
• snd_pcm_oss
• snd_seq_midi
• snd_seq_oss

Because snd_hda_intel's extensive dependency chain, loading of snd_hda_intel therefore results in several modprobe calls and is the reason 3 modprobe instances are running around second 16 in the bootchart.

To get a sense of the impact that udev's modprobe calls have on boot time, we temporarily disabled the rules in udev's 90-modprobe.rules. This change decreased the boot time by 25 seconds and resulted in a dramatically simpler bootchart. Of course, disabling the loading of all modules is not actually a viable solution as some are actually needed during boot.

Stats

Device event stats generated by running script on udev log. The log was from an image with no boot-time speed-up changes.

Module load stats generated by running script on udev log. The log was from an image with no boot-time speed-up changes.

Summary of Solution

We created an executable in /sbin called delayonstartup that a udev rule can pass its RUN command to. If the system is not starting up, delayonstartup will run the command. If the system is starting up, delayonstartup will add the command to a file that will be run after startup has completed. We pass the /sbin/modprobe command, along with its parameters, to delayonstartup. However, delayonstartup can theoretically be used with any udev command specified by the RUN action.

The delayonstartup executable knows whether the system is starting up or not based on whether a poststartupcmds file exists in the /dev/.udev directory. The udev init script in /etc/init.d/ will create this file before it calls udevtrigger to trigger udev events on cold boot. The init script creates the file in /dev/.udev as the read-write portion of the rootfs is not yet available during this stage of initialization. This file is also the file that delayonstartup writes the delayed commands to.

The udev-finish init script runs after the udev init script has completed as well as after the read-write portion of the rootfs has become available. This script moves udev files generated in the /dev directory during early startup to their proper place in the rootfs. Originally, this script just moved the .udev.log file from /dev/ to /var/log/udev/ (and renamed it udev) and moved any udev rule files generated in /dev/.udev/ to /etc/udev/rules.d/. We now also have the script move the poststartupcmds file from /dev/.udev/ to /etc/udev/.

To be able to run the commands saved to poststartupcmds, we created an executable in /sbin/ called runpoststartupcmds that sources poststartupcmds. We trigger the runpoststartupcmds executable after the UI is up by creating a runpoststartupcmds.desktop file in /usr/share/autostart/ that specifies runpoststartupcmds in its Exec line. The Exec lines in each of the files in /usr/share/autostart/ are executed in the background by the ume-gui-start script in /usr/share/ume-config-common/ before starting the hildon-desktop.

Implementation Details and Questions

Reduced time from cold boot to UI interaction by 8 seconds.

Associated bootchart.

Changes to /etc/udev/rules.d/90-modprobe.rules:

• Added rules to explicitly match devices that are part of the acpi, input, serio, and scsi subsystem. These rules run modprobe if the device has a kernel-supplied module alias. These rules are needed as the kernel modules for these devices are required during early boot. If, in the future, other modules need to be loaded during early boot, new rules for these modules should be added. Not loading modules during early boot by default and requiring a rule to be explicitly written for a module if it should be loaded during early boot will hopefully force us to have a tighter boot process.

• Changed the existing rule that runs modprobe for any device with a kernel-supplied module alias from:

  • ENV{MODALIAS}=="?*",                   RUN+="/sbin/modprobe -Q $env{MODALIAS}" to:

    ENV{MODALIAS}=="?*",                   RUN+="/sbin/delaycmdonstartup /sbin/modprobe -Q $env{MODALIAS}"

runpoststartupcmds:

• To ensure that the commands saved to poststartupcmds do not run until the UI is up, runpoststartupcmds first sleeps for 10 seconds before sourcing the poststartupcmds file. QUESTION: Is this acceptable? Is there somewhere else other than from /usr/share/autoshare/ that we can trigger runpoststartupcmds so we do not need to sleep and guess when it is okay to source the poststartupcmds file? COMMENT: upstart task may be the answer.

runpoststartupcmds.desktop:

• By the time ume-gui-start runs (the script that iterates through the .desktop files in /usr/share/autoshare/), we are no longer running as root. Therefore, runpoststartupcmds.desktop must use sudo when executing /sbin/runpoststartupcmds. QUESTION: Is this acceptable? Should we set the uid bit for runpoststartupcmds? NOTE: The reason runpoststartupcmds must run as root is because it may be sourcing commands such as modprobe (which need to be run as root). COMMENT: upstart task may be the answer.

Modules that are now loaded after the UI is up (NOTE: Need to do more testing to ensure the delayed loading of these modules does not break any functionality):

• soundcore
• snd
• snd_seq_device
• snd_timer
• snd_hwdep
• snd_page_alloc
• snd_seq
• snd_seq_midi_event
• snd_pcm
• snd_rawmidi
• snd_mixer_oss
• snd_pcm_oss
• snd_seq_midi
• snd_seq_oss
• iusbc
• sdhci
• pata_acpi
• i2c_core
• i2c_sch
• cdc_acm
• v4l1_compat
• videodev
• compat_ioctl32
• uvcvideo
• bluetooth
• hso
• v4l2_common
• hci_usb

Other Solutions Considered

Call udevtrigger Twice with Specific Events Suppressed

udevtrigger supports the following parameters (from udevtrigger man page):

• --subsystem-match=subsystem
  • Trigger events for devices which belong to a matching subsystem. This option can be specified multiple times and supports shell style pattern matching.
  --subsystem-nomatch=subsystem
  • Do not trigger events for devices which belong to a matching subsystem. This option can be specified multiple times and supports shell style pattern matching.
  --attr-match=attribute=value
  • Trigger events for devices with a matching sysfs attribute. If a value is specified along with the attribute name, the content of the attribute is matched against the given value using shell style pattern matching. If no value is specified, the existence of the sysfs attribute is checked. This option can be specified multiple times.
  --attr-nomatch=attribute=value
  • Do not trigger events for devices with a matching sysfs attribute. If a value is specified along with the attribute name, the content of the attribute is matched against the given value using shell style pattern matching. If no value is specified, the existence of the sysfs attribute is checked. This option can be specified multiple times.

We could limit the device events triggered during early boot by using these parameters and trigger the suppressed events after the UI is up by calling udevtrigger again with the complement of the parameters originally passed. For example, if --subsystem-nomatch=pci was passed to udevtrigger on early boot, --subsystem-match=pci would be passed to udevtrigger on late boot. We could specify which subsystems and attributes to skip during early boot (and then trigger on late boot) in a configuration file so we would not have to hard code this information in the initialization scripts that call udevtrigger.

Unfortunately, this approach had several drawbacks that prevented it from being chosen as the solution. The following are the main drawbacks:

• We end up iterating through sysfs twice looking for device events to trigger. This is obviously inefficient.
• The match/nomatch parameters only provide broad control over which device events are triggered. An event is triggered if the device matches just one of the "match" parameters and is not triggered if it matches just one "nomatch" parameter. Triggering an event if the device matches several attributes as well as a specific subsystem is currently not supported.
• When a device matches a nomatch parameter, no rules for the device are executed (since the device event itself is not generated). In other words, we do not have the ability to only disable one or two rules for an event. In our case, we would like all but modprobe rules to be executed.

Manually Load All Modules

For devices running a customized UME image, we can take more drastic steps. During boot, we can completely disable any udev rules that call modprobe and rely on an init script that uses insmod to manually load all modules needed during early boot. This will likely produce the largest boot-time speed-up as only what is absolutely needed during boot is loaded. We have complete control over the modules loaded. The main drawback of this approach, however, is that the init script will be difficult to maintain moving forward. Note that the rules must be re-enabled after boot so hot-plugging will work properly.

The following is a bootchart for one such customized UME image. The change reduced the boot time by almost 10 seconds.

Disable udevsettle on Boot

On cold boot, udevsettle is called after udevtrigger has completed. udevsettle waits until all the events triggered by udevtrigger have been handled. It seems unfortunate that we must wait for all the events to be handled before our remaining system initialization can proceed. In most cases, the call to udevsettle can be safely removed to further reduce the boot time.