= Reducing Boot Time of UME Image = == Summary of Problem == Looking at the [[attachment:unmodified_boot_3_20.png|bootchart]] for a UME image, you can see that modprobe runs for a significant portion of the boot process. Around second 16, there are even 3 modprobe instances running concurrently. udev has a rule that calls modprobe anytime it receives a kernel uevent for a device with an associated module alias. This rule is the last rule in `/etc/udev/rules.d/90-modprobe.rules`. On cold boot, `/etc/rcS.d/S10udev` calls udevtrigger to trigger a kernel device uevent for all devices. This results in modprobe being called for all devices with a module alias--even if the device's module is not needed to boot. The snd_hda_intel is one such module and is particularly troublesome because it causes all of the following sound-related modules to be loaded: * soundcore * snd * snd_seq_device * snd_timer * snd_hwdep * snd_page_alloc * snd_seq * snd_seq_midi_event * snd_pcm * snd_rawmidi * snd_mixer_oss * snd_pcm_oss * snd_seq_midi * snd_seq_oss Because `snd_hda_intel`'s extensive dependency chain, loading of snd_hda_intel therefore results in several modprobe calls and is the reason 3 `modprobe` instances are running around second 16 in the bootchart. To get a sense of the impact that udev's `modprobe` calls have on boot time, we temporarily disabled the rules in udev's `90-modprobe.rules`. This change decreased the boot time by 25 seconds and resulted in a dramatically [[attachment:no_modprobe_boot_3_20.png|simpler bootchart]]. Of course, disabling the loading of all modules is not actually a viable solution as some are actually needed during boot. === Stats === [[attachment:deviceEventStat_3_31.txt|Device event stats]] generated by running script on udev log. The log was from an image with no boot-time speed-up changes. [[attachment:moduleLoadStats_3_31.txt|Module load stats]] generated by running script on udev log. The log was from an image with no boot-time speed-up changes. == Summary of Solution == We created an executable in `/sbin` called `delayonstartup` that a udev rule can pass its RUN command to. If the system is not starting up, `delayonstartup` will run the command. If the system is starting up, `delayonstartup` will add the command to a file that will be run after startup has completed. We pass the `/sbin/modprobe` command, along with its parameters, to `delayonstartup`. However, `delayonstartup` can theoretically be used with any udev command specified by the RUN action. The `delayonstartup` executable knows whether the system is starting up or not based on whether a `poststartupcmds` file exists in the `/dev/.udev` directory. The `udev` init script in `/etc/init.d/` will create this file before it calls `udevtrigger` to trigger udev events on cold boot. The init script creates the file in `/dev/.udev` as the read-write portion of the rootfs is not yet available during this stage of initialization. This file is also the file that `delayonstartup` writes the delayed commands to. The `udev-finish` init script runs after the `udev` init script has completed as well as after the read-write portion of the rootfs has become available. This script moves udev files generated in the `/dev` directory during early startup to their proper place in the rootfs. Originally, this script just moved the `.udev.log` file from `/dev/` to `/var/log/udev/` (and renamed it `udev`) and moved any udev rule files generated in `/dev/.udev/` to `/etc/udev/rules.d/`. We now also have the script move the `poststartupcmds` file from `/dev/.udev/` to `/etc/udev/`. To be able to run the commands saved to `poststartupcmds`, we created an executable in `/sbin/` called `runpoststartupcmds` that sources `poststartupcmds`. We trigger the `runpoststartupcmds` executable after the UI is up by creating a `runpoststartupcmds.desktop` file in `/usr/share/autostart/` that specifies `runpoststartupcmds` in its Exec line. The Exec lines in each of the files in `/usr/share/autostart/` are executed in the background by the `ume-gui-start` script in `/usr/share/ume-config-common/` before starting the hildon-desktop. === Implementation Details and Questions === Reduced time from cold boot to UI interaction by 8 seconds. Associated [[attachment:reducedModProbe_4_1.png|bootchart]]. Changes to `/etc/udev/rules.d/90-modprobe.rules`: * Added rules to explicitly match devices that are part of the acpi, input, serio, and scsi subsystem. These rules run `modprobe` if the device has a kernel-supplied module alias. These rules are needed as the kernel modules for these devices are required during early boot. If, in the future, other modules need to be loaded during early boot, new rules for these modules should be added. Not loading modules during early boot by default and requiring a rule to be explicitly written for a module if it should be loaded during early boot will hopefully force us to have a tighter boot process. * Changed the existing rule that runs `modprobe` for any device with a kernel-supplied module alias from: `ENV{MODALIAS}=="?*", RUN+="/sbin/modprobe -Q $env{MODALIAS}"` to: `ENV{MODALIAS}=="?*", RUN+="/sbin/delaycmdonstartup /sbin/modprobe -Q $env{MODALIAS}"` `runpoststartupcmds`: * To ensure that the commands saved to `poststartupcmds` do not run until the UI is up, `runpoststartupcmds` first sleeps for 10 seconds before sourcing the `poststartupcmds` file. QUESTION: Is this acceptable? Is there somewhere else other than from `/usr/share/autoshare/` that we can trigger `runpoststartupcmds` so we do not need to sleep and guess when it is okay to source the `poststartupcmds` file? COMMENT: upstart task may be the answer. `runpoststartupcmds.desktop`: * By the time `ume-gui-start` runs (the script that iterates through the .desktop files in `/usr/share/autoshare/`), we are no longer running as root. Therefore, `runpoststartupcmds.desktop` must use `sudo` when executing `/sbin/runpoststartupcmds`. QUESTION: Is this acceptable? Should we set the uid bit for `runpoststartupcmds`? NOTE: The reason `runpoststartupcmds` must run as root is because it may be sourcing commands such as `modprobe` (which need to be run as root). COMMENT: upstart task may be the answer. Modules that are now loaded after the UI is up (NOTE: Need to do more testing to ensure the delayed loading of these modules does not break any functionality): * soundcore * snd * snd_seq_device * snd_timer * snd_hwdep * snd_page_alloc * snd_seq * snd_seq_midi_event * snd_pcm * snd_rawmidi * snd_mixer_oss * snd_pcm_oss * snd_seq_midi * snd_seq_oss * iusbc * sdhci * pata_acpi * i2c_core * i2c_sch * cdc_acm * v4l1_compat * videodev * compat_ioctl32 * uvcvideo * bluetooth * hso * v4l2_common * hci_usb == Other Solutions Considered == === Call udevtrigger Twice with Specific Events Suppressed === udevtrigger supports the following parameters (from udevtrigger man page): --subsystem-match=subsystem Trigger events for devices which belong to a matching subsystem. This option can be specified multiple times and supports shell style pattern matching. --subsystem-nomatch=subsystem Do not trigger events for devices which belong to a matching subsystem. This option can be specified multiple times and supports shell style pattern matching. --attr-match=attribute=value Trigger events for devices with a matching sysfs attribute. If a value is specified along with the attribute name, the content of the attribute is matched against the given value using shell style pattern matching. If no value is specified, the existence of the sysfs attribute is checked. This option can be specified multiple times. --attr-nomatch=attribute=value Do not trigger events for devices with a matching sysfs attribute. If a value is specified along with the attribute name, the content of the attribute is matched against the given value using shell style pattern matching. If no value is specified, the existence of the sysfs attribute is checked. This option can be specified multiple times. We could limit the device events triggered during early boot by using these parameters and trigger the suppressed events after the UI is up by calling `udevtrigger` again with the complement of the parameters originally passed. For example, if `--subsystem-nomatch=pci` was passed to udevtrigger on early boot, `--subsystem-match=pci` would be passed to `udevtrigger` on late boot. We could specify which subsystems and attributes to skip during early boot (and then trigger on late boot) in a configuration file so we would not have to hard code this information in the initialization scripts that call `udevtrigger`. Unfortunately, this approach had several drawbacks that prevented it from being chosen as the solution. The following are the main drawbacks: * We end up iterating through sysfs twice looking for device events to trigger. This is obviously inefficient. * The match/nomatch parameters only provide broad control over which device events are triggered. An event is triggered if the device matches just one of the "match" parameters and is not triggered if it matches just one "nomatch" parameter. Triggering an event if the device matches several attributes as well as a specific subsystem is currently not supported. * When a device matches a nomatch parameter, no rules for the device are executed (since the device event itself is not generated). In other words, we do not have the ability to only disable one or two rules for an event. In our case, we would like all but modprobe rules to be executed. === Manually Load All Modules === For devices running a customized UME image, we can take more drastic steps. During boot, we can completely disable any udev rules that call `modprobe` and rely on an init script that uses insmod to manually load all modules needed during early boot. This will likely produce the largest boot-time speed-up as only what is absolutely needed during boot is loaded. We have complete control over the modules loaded. The main drawback of this approach, however, is that the init script will be difficult to maintain moving forward. Note that the rules must be re-enabled after boot so hot-plugging will work properly. The following is a [[attachment:insmodandNo90ModprobeRules_4_17.png|bootchart]] for one such customized UME image. The change reduced the boot time by almost 10 seconds. === Disable udevsettle on Boot === On cold boot, `udevsettle` is called after `udevtrigger` has completed. `udevsettle` waits until all the events triggered by `udevtrigger` have been handled. It seems unfortunate that we must wait for all the events to be handled before our remaining system initialization can proceed. In most cases, the call to `udevsettle` can be safely removed to further reduce the boot time.