Debugging udev and some related hints and tips

Verbose output

The main approach to debugging a problem that might be related to udev is to run udevd --verbose. The output is sent to the terminal and also to the system console via syslog; unfortunately the default configuration does not save the udev output to a file anywhere.

The generated debug information gives information about the incoming events and details of the resulting external commands run by udev and their output. Additionally, using udevmonitor can be helpful.

In the initramfs, specifying the break=top or break=premount as a boot argument will interrupt the boot and allow udev to be run by hand.

Races and similar problems

Nearly everything done with udev involves a risk of races.

When debugging programs invoked from udev it is usually necessary to properly understand the programs' locking and concurrency behaviour.

The kernel events processed by udev do not always correspond to the high-level events that one might expect; in particular, some devices generate one event on creation but are not properly useable or sane until a second `change' event.

For these reasons it is best if the programs run from udev scripts are idempotent and error-tolerant. Making them idempotent also permits the use of the `watershed' utility which can optimise away redundant calls.

When the creation of a device is the result of another userland activity (for example, the use of dmsetup), there is no way provided by udev to resynchronise the initiating process. This leads to races between the initiating process and the udev event processing. Unfortunately there is not currently any good answer to this. Some existing ad-hoc approaches are described in UdevLvm and UdevDeviceMapper.

Unfortunately enabling verbose output can sometimes change the timings enough to eliminate a particular race and there is not currently a good workaround for this problem. There is no way to suppress output to syslog (or to the console in the initramfs). Obviously it is best in any case to thoroughly design and review the setup to ensure that it has no races, since being able to reproduce a race before and but not after some change does not give any great confidence that the bug is fixed.

Other quirks to be aware of

Although udev commands are specified as strings, they are not executed via a shell.

When no absolute path is specified, udev executes only programs in /lib/udev.

If a rule has more than one RUN+= consequence, only one takes effect.

If the udevd from the initramfs is not killed before switching to the real root fs, then the real root fs may not start its udevd because one is already running. Generally, when messing about in the initramfs it is necessary to pkill udevd before exiting to the main boot sequence.

Failures to start udevd in the main system typically result in bizarre faults such as incorrect permissions on, or complete absence of, /dev/null or other critical /dev entries.