DesktopTeam/Specs/HardyHardwareDetection

HardyHardwareDetection

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

Launchpad Entry: hardy-hardware-detection
Packages affected: hal, cryptsetup, consolekit, udev, discover, xorg

Summary

In the last years, upstreams have provided vast improvements in hardware detection infrastructure. We review which parts of Debian/Ubuntu specific hardware detection bits are obsolete and how to consolidate and update them.

Release Note

This specification involves a lot of smaller and larger robustifications and changes, but these mostly fall under the "bug fix" and "make it work as it is supposed to" category, and thus are not really worth mentioning in release notes.

Rationale

Remove maintenance in Debian/Ubuntu, use upstream's solution and collaborate on that instead.
Much more dynamic, supports future hardware, too.
Much more robust, since removing overhead.

Use Cases

Jean-Luc buys a new Canon digicam, which uses the PtP protocol. At the time of Hardy release this was not yet known, but after plugging it in he can access it with the normal photo tools without any permission issues.
Will has used an nVidia graphics card so far, but now buys a new one from ATi. After installing it, he boots Ubuntu and does not need to figure out how to fix the broken X.org (as he had to do in Gutsy).
Geordi uses the proprietary nVidia graphics drivers. The other day he installs a newer custom kernel, but the nVidia driver does not yet work on this kernel. X.org automatically falls back to nv.
Beverley buys a new Dell Latitude laptop which has a bug in its ACPI implementation. The power management recognizes the model and enables a quirk workaround which automatically makes suspend to RAM work flawlessly.

Design

Device ownership: Make removable USB devices (scanners, cameras, external hard disks, etc.) accessible to the user on the current foreground console instead of providing static udev rules and using group membership based access control. Then the current set of fine-tuned udev rules and large lists of autogenerated udev rules for model-based vendor/product ID matching can disappear entirely, and the access control will work for e. g. future scanners or cameras, too.
X.org configuration: Drop the vast and hard to maintain set of scripts, discover, and current debconf implementation in favor of using and improving current X.org's built-in autodetection, do not use an xorg.conf in general and only use it for overriding specific settings. Still provide a debconf interface for tweaking xorg.conf, but do it the right way this time ("debconf is not a database").
Power management: Replace ubuntu specific powermanagement-interface and upstream-unmaintained acpi-support with pm-utils.
Drop our current legacy kernel options to avoid some race conditions and uncover legacy code which is error prone.

Implementation

Device ownership

Drop the default system groups plugdev and scanner. Enable ConsoleKit support in hal, which assigns read/write permission to removable USB devices with dynamic ACLs.
As a followup to above, drop the long vendor/product ID based udev rules of libgphoto and libsane. Ensure that their preinsts remove the obsolete conffiles on upgrade.
Drop the default system groups netdev and powerdev, drop our in-house development libpam-foreground, drop the network-manager and gnome-power-manager patches to query libpam-foreground. Instead, do the "user is in foreground session" test with PolicyKit rules.

X.org configuration

Aim for fully automatic and dynamic configuration of X.org so that usually xorg.conf is not needed at all. This provides the following advantages:

Fully utilize upstream's vastly improved hardware autodetection.
Be robust with hardware changes.
Provides built-in failover in case a driver stops working for some reason (update, kernel ABI break, proprietary driver update, etc.)
Can automatically use newly installed drivers without manual configuration (e. g. synaptics or nvidia-glx).
Changes in driver policy can be made easier, without the need of changing xorg.conf on upgrades.
xorg.conf will only contain the necessary custom bits and avoids boilerplate.

Notes on particular sections:

Files: File locations are determined by distribution packages, and should have the right defaults. This particularly affects font packages. Gutsy does not configure any explicit font paths any more, and yet xlsfonts proves that the server finds them, so the section can be dropped entirely.
- DONE - This is now essentially done, the server uses built-in paths for fonts, and the section is not generated anymore. TimoAaltonen
InputDevice: X.org server 1.4 now supports dynamic input device detection from hal, the input sections can go away. One particular bit that still needs to be configured is the default keyboard layout, which should be done in a hal FDI (this is already supported). (Note that relying on GNOME to set up the keyboard layout is not sufficient: first, it does not apply to gdm, and second other window managers do not provide their own keyboard layout handling.)
Device: The X.org server automatically detects the driver to be used based on available hardware and available drivers if this section is missing. However, the preference list still needs some tweaks (e. g. it should be nvidia > nv > vesa, but currently it prefers nv over nvidia). When special options need to be passed to the driver, the section must remain, though. This means that the discover package and a large amount of detection scripts can be dropped.
Monitor: Again, the X.org server's built in monitor detection usually does a better job than the detection in the Debian/Ubuntu scripts.
Screen: References to device and monitor are implicit, default monitor resolution is detected automatically, and resolutions and multihead are better configured with XRandR and displayconfig-gtk.
ServerLayout: This only contains references to the previous sections, which are implicit when not given.
- DONE - No longer written. TimoAaltonen

Debconf interface: For manual tweaks and situations where X.org does not start at all, we should still provide a lightweight debconf interface. However, this should use libxf86config, pyxf86config, or guidance-backends and thus respect the settings in the existing xorg.conf instead of clobbering it with the debconf values and abusing debconf as a database.

Debian is about to do the same changes for X.org, and their and Ubuntu's X.org maintainers are working together on this.

UPDATE: As of the 2007-12-18 xorg sync with Debian, the above is all largely done now.

X.org configuration refactoring

Principally, to make more use of X.org's built-in autodetection, we simply need to remove our own infrastructure. However, as there may be cases where we wish to retain mechanisms for overriding the autodetection, we need to do this in an organized fashion.

discover: The remaining useful pieces in discover is the pciid-to-driver list. David Nusinow (Debian) has code to maintain such a list within the xserver, which perhaps could be used for this purpose. As well, many of our pciid-to-driver mappings may no longer be necessary, due either to being carried in the x drivers themselves, or to the original issue being resolved; so the whole list needs to be re-analyzed, and hopefully the bulk can be either dropped or pushed up to the respective drivers. Even if we can get rid of *all* of our own overrides, we should still retain this override mechanism, as it will provide a method for updating X.org's pciids post-release, without needing to rebuild the xserver. discover can then be removed from the X.org configuration system by removing the calls to it from xserver-xorg.postinst.in (e.g. lines ~800 to 875, ~997).
sparc customizations: xserver-xorg.postinst.in and other scripts contain some logic that is particular to sparc, that may not be covered adequately by upstream's xorg autoconfiguration code. If we wish to retain this level of sparc support, the logic will need to be moved to more appropriate places, if it's not there already. (This is a good example of Ubuntu/Debian-specific configuration logic that really needs to be pushed upstream, so the maintenance burden is not left to Debian and us.)
xresprobe: This script turned out to be the cause of a vast number of resolution detection issues that had plagued Feisty. For Gutsy, we succeeded at resolving the vast majority of its bugs (in Hardy, it's nicely rare to have resolution configuration issues), however it still has problems (such as when monitor EDID info can't be read) and besides, it's obsolete. This can be removed by disabling the xresprobeint call from xserver-xorg.postinst.in (around line 2035). There are some special cases that have been wrapped in either inside xresprobe or in the code surrounding the call to it in xserver-xorg.postinst.in; this will need to be moved to the drivers or other more appropriate locations.
- DONE: xresprobe is now no longer used by dexconf.
dexconf: In generating the xorg.conf, this script will need to be modified to not print out sections if the information the section needs is not defined. This includes Screen, Monitor, InputDevice, etc. The ServerLayout section will also need logic to skip referencing sections that were not written out.
- DONE: dexconf has now received these changes
video driver selection: xserver-xorg.postinst.in uses discover to learn about the graphics hardware. In the general case, this can now be handled automatically by X, however there are special cases to take care of various corner cases and for setting options like defaultdepth. Some of these things are obsolete and can be dropped, however others will need to be retained but moved to more appropriate locations such as into the drivers themselves.
keyboard selection: This is currently done in xserver-xorg.postinst.in, starting around line 1138. This currently configures the keyboard first by looking if the keyboard layout is specified in the console configuration (/etc/default/console-setup), then by selecting from debian-installer/keymap. With this info, it then looks up the corresponding keyboard rules files and keycodes. It also contains a few hardware-specific workarounds. The new way of doing things is to rely on evdev, which is more flexible and a better design, and permits per-user default settings via Xi and XKB requests at login time; it is used by default if no keyboard section is specified in xorg.conf, thus xserver-xorg.postinst.in simply needs to be modified to not define keyboard parameters by default, but to instead write them into the x11-input.fdi file if needed (and/or console-setup). The hardware-specific workarounds need to be moved to hal and/or evdev, and we need to ensure that all our keymap-to-rules mappings are covered as well. Once this is done, the corresponding keyboard config code in xserver-xorg.postinst.in should be dropped; however, the user should still be able to override this selection and specify a different keyboard by running dpkg-reconfigure --priority=high xserver-xorg.
mouse protocol selection: For the mouse protocol, it's similar to keyboard - we pretty much just need to change DEFAULT_PROTOCOL to be empty in xserver-xorg.postinst.in for all of the port types it supports, instead of "PS/2", et al. Dexconf then needs to be made to not write this section if the protocol is undefined. The xserver will then use "evdev" by default. Also, the priority_ceil can probably be raised from "low" to "medium".
other input devices: There are a few devices being carried in the Debian/Ubuntu config files, which should either be dropped or moved upstream. For instance, vmmouse and wacom tablets. Moving them upstream will open better chances for them to benefit from hotplug autodetection, and will eliminate having to maintain these customizations ourselves.
xserver-xorg.postinst.in cleanup: In addition to the required changes mentioned above, there are other hardware-detection sections in this script that will need to be reviewed and, likely, dropped. As part of the Hardy development effort, this entire script should be reviewed and refactored in conjunction with Debian, with the objective of reducing its length from 2100 lines to something more manageable and maintainable (200 lines would be a nice goal). This should be achieved by (in order of preference): a) eliminating obsolete code now handled in xserver; b) moving logic out of this config script and into the xserver; c) moving logic into other existing configuration tools where it will be more appropriate; d) breaking logic out into independent scripts, that could be of use post-install; e) using plain-text config files, which at least would be simpler for people with limited Xorg knowledge to update/maintain.
- UPDATE: xserver-xorg.postinst.in's length has been reduced by about half, to 1196 lines.
gdm: None of the above appears to be likely to cause conflicts with gdm, however based on past experience, we'll need ample testing here to be certain.
- UPDATE: The bulk of the xorg changes are now uploaded in time for alpha-2, enabling us to begin testing as soon as that's out.

Power management

pm-utils is discovered along with HAL on freedesktop.org, provides a distribution independent power management backend, and is the place to collect all hardware specific quirks. We should make use of this and contribute our changes upstream instead of continuing to maintain the deprecated acpi-support and the Ubuntu specific powermanagement-interface abstraction.

Starting from Hal 0.5.10, the only supported backend is pm-utils.

Kernel configuration

CONFIG_USB_DEVICE_CLASS=n: This drops /sys/class/usb_device/ which is subject to race conditions when using libusb. With 'n' they get replaced by additional attributes to the actual USB device.
CONFIG_SYSFS_DEPRECATED=n: Get rid of the /class hierarchy so that all data will be under /devices where it should be.
CONFIG_TMPFS_POSIX_ACL=y: Required for hal's ConsoleKit integration to set ACLs on device nodes.
Sync our udev rules with upstream's for the corresponding userspace side.

Test/Demo Plan

To be done when a beta is available.

Outstanding Issues

Would be nice to have something like sysfs for various X parameters (drivers, pciid->driver mappings, etc.) Much of this info can be gathered via various disparate tools (e.g. xdpyinfo, xdriinfo, xrandr...) however a unified api would give this all in one go and would be much easier. This would be beneficial for programs like restricted-manager and displayconfig-gtk.

Ubuntu Wiki