OptionsEditor

  • Launchpad Entry: xorg-options-editor

  • Created: 2008-06-02

  • Contributors: bryceharrington, tseliot

  • Packages affected: x-kit

Summary

Implements an xorg.conf editor for adding/removing/changing driver and server options. This is done in two phases: The first phase simply establishes the foundations (parser, backend, etc.) and provides for setting the "Virtual" Option. The second phase implements the full GUI editing capability.

Rationale

Xorg's configuration system has matured greatly in recent years, to the point that editing xorg.conf is usually unnecessary: the X server autodetects pretty much everything, and distro install scripts can largely take care of the remainder. With the introduction of Xrandr, many common configuration tasks (like setting resolutions) that used to be done by editing xorg.conf and restarting can now be done at run-time using GUI tools like Screen Resolution.

However, there are still many little corner cases where the user will need or wish to make adjustments to X that can currently only be done in xorg.conf; largely, these cases involve setting one or more Options either in a Device section or in the ServerFlags section. For example, we may wish them to turn off certain things for testing purposes, or they may wish to experiment with settings to attain better performance or stability than they get with the defaults.

A GUI tool makes these settings more accessible to users, which will make Ubuntu's users better able to participate in troubleshooting X problems and in getting the most out of their systems.

Scope

This is not a general purpose "fix broken X" tool. In many cases where setting Options is required to fix X, it's needed because X won't start up at all, and therefore the user isn't in a GUI environment anyway. Thus this spec isn't concerned with providing functionality for selecting the driver, altering monitor settings, and so on.

This is not considered to be a tool that very many users will need to use, and thus is intended to be available only via an "Advanced..." button or the like.

It is not a tool for use in selecting the driver to use for a given graphics card - this is a duty for Jockey or other X configuration tools to handle. We focus only on options in this tool.

The man page parsing logic contained in this tool will be specific to X.org man pages. These pages share a formatting style that differs from other man pages; the parsers will make use of this formatting style for parsing purposes.

Use Cases

  • Terri wishes to set up a dual-head display on Intrepid, but this requires setting Option "Virtual" "3840 1200" in xorg.conf, which she is not comfortable editing. Instead she uses the X Options Editor to specify it.

  • Sarah reads in a bug report that using Option "VideoKey" "0x01" resolved a problem very similar to hers. She uses X Options Editor to add this option, restarts, and can easily test it out.

  • Leann wishes to report an X bug, but reads that she should enable Option "NoTrapSignals"  "true" first so she collects correct debug info.

  • Kirsten likes having ctrl+alt+backspace exit her system immediately. She switches it on by toggling the Option "DontZap" "true" option.

Design

The editor's overall architecture will be split into frontend and backend modules, with the idea of sharing the backend with other tools which need to modify xorg.conf. The philosophy is that the fewer chunks of code we have that are modifying xorg.conf, the more consistent the result will be. Also, obviously this allows sharing code maintenance of this part. (As an example, we could use the same parser/writer lib with Jockey and in the installer.)

The GUI editor itself will be composed of a list of xorg.conf sections with options available for modification (e.g. Device and ServerFlags). Selecting a section will display a table of options and their values that the user can modify. On hitting Apply, the xorg.conf will be backed up and modified, and the user prompted to restart X for the changes to take effect.

Phase 1: Foundations + Option "Virtual"

The first portion of the project will focus on a single goal: Providing a mechanism to add "Virtual" option settings to xorg.conf without requiring the user to edit that file directly. Ideally, this should be hooked into the Screen Resolution tool to set the option automatically when the user chooses a dual head layout beyond the limits of the current virtual framebuffer size.

This means the focus will be first on the implementation of the backend xorg.conf parser/writer, not on the GUI editor.

Phase 1 is to be delivered for Intrepid by Alpha 1. Depending on the complexity of its implementation, it may be appropriate to SRU this feature to Hardy as well.

UPDATE: the Python part is complete. Here it is possible to see a screencast of the Python part (still WIP) in action. The screencast doesn't reflect the current status however it gives an idea of how this will work in practice. The Python script can be called with:

python multiscreen.py 0,0:1024x768 1024,0:1280x1024 0,768:1024x768

i.e. "x_position","y_position":"h_resolution"x"v_resolution"

Phase 2: GUI Editor

The remainder of the work to implement the GUI will be done in the second phase, and is a much lower priority than phase 1.

Phase 2 is to be delivered by Jaunty's feature freeze date.

The UI should be laid out in a three-pane view similar to the Add/Remove Applications window. The left-most window is a tree-view for selecting the device to configure; the top selectable item is the Server Flags section. The top right pane is a list of options to set, with checkboxes to enable them, the current or default value, and a short 1-sentence summary. The bottom right pane is a detailed view of the option, with a complete description.

A "Test" button should be available, which saves the settings to a temporary config file, and attempts to launch an X server on a new VT, with a revert dialog. (See displayconfig-gtk for some sample python/C code for implementing all of this).

The "Apply" button should actually say "Save" (since you have to restart X).

The Safety Net - Backup and Restore

Many of the options we wish to make available to the user are dangerous. They can leave the user's system in a wide variety of error states: Crashed, hung, "No Signal", unstable, unusably slow, with the wrong resolution, or more. Because the options are written to xorg.conf to take effect, they will persist across reboots. It is therefore imperative that we ensure that any changes made can be undone. Indeed, the reversibility is one of the primary requirements and features of this tool.

In the ideal case, the user can test out settings before applying them permanently. There are two mechanisms for achieving this:

  1. Write the changes to a temporary xorg.conf and start a second X session with a dialog prompt "Do you want to keep these settings?" which only saves the xorg.conf as the new default if confirmed.
  2. Write the changes to a temporary xorg.conf and restart the session one time. If it comes up properly, mark the temporary xorg.conf as good and save it as the new default. On the second boot of the session, the original xorg.conf should be used, and the temporary one marked bad.

HINT: we could keep diff files created against an original (very stripped down) xorg.conf which (both the original xorg.conf and the diffs) we would store in a separate directory.

Inevitably though, the user will at some point inadvertently end up saving a bad xorg.conf.

There are several entry points a user with a ruined X configuration might find themselves:

  • LiveCD
  • grub (kernel recovery mode / ala xfix)
  • Bulletproof-X mode
  • gdm login screen
  • Regular session, but with undesired configuration

In each of these cases, a tool must be available with the capability to do the following things:

  • Revert to the last known good xorg.conf
  • Restore to "Factory Presets" (ala dpkg-reconfigure -phigh xserver-xorg)

It is expected that users of this tool will wish to try a variety of different options, some of which result in viable X sessions, others which don't. Beyond merely backing up the old xorg.conf, we need a more sophisticated system for managing the backups. A command line tool (preferably in bash or something equally quick to run) will be used to interface to the backup collection, which has the following features:

  • Snapshots a "Hardware Layout" associated to an identifier, generated by these commands:
    • lspci -vvnn | grep -A1 "VGA compatible"
    • xinput --list
    • xrandr --verbose | grep -A8 EDID_DATA
  • For each xorg.conf managed by the system, track the following data:
    • nickname
    • md5sum of xorg.conf
    • md5sum of valid hardware layout(s) to use with this config (or blank to apply to all)
    • creation date
    • last successful boot date/time
    • last unsuccessful boot date/time
    • quantity of successful boot attempts
    • quantity of unsuccessful boot attempts
    • errors (and maybe warnings?) encountered from last boot attempt
  • The command line tool is invoked during each X boot at several points, so that it may note the success/failure of each attempt:
    • Prior to attempting to start X
      • Check the hardware layout. If the current xorg.conf is not identified as supporting this layout and a different xorg.conf entry does, switch to that xorg.conf. (See LP #257825)
    • When the login screen has successfully displayed
    • When a user session is being started
    • When a user session has been successfully displayed
  • The command line tool can be invoked by other tools or directly by the user to:
    • List known hardware layouts and their identifier
    • Associate a hardware layout identifier with an xorg.conf configuration
    • Mark one configuration "preferred" for the specified hardware layout
    • List the configurations being managed
      • Filter to display:
        • All
        • Only ones marked "preferred"
        • Those that have successfully booted 100%
        • Those that have successfully booted at least some time
        • Those that have always failed to boot
        • Those that have failed at least some time
      • Sort by:
        • Age
        • Time since last successful boot
        • Number of successful boots
        • Ratio of success/failure boots
    • Add a new xorg.conf to manage
    • Remove a xorg.conf from being managed (and delete it from the filesystem)

Follow-on Work / End of Life

This is not intended to be a general purpose configuration tool, so if strong use cases turn up that require use of this tool, we should consider spinning off special-purpose tools (like the wacom config tool described in a separate spec) to focus on those use cases. This options editor should always remain an under-the-hood tool of last resort.

The screen-configuration-ui specification describes a future follow-on and/or replacement for this tool, to be targeted for the Jaunty+1 or later timeframe. So we anticipate that ultimately this tool (or at least its functionality) will be subsumed into another tool.

There has also been discussion upstream regarding making driver/server configuration options dynamically settable at runtime (i.e. without editing xorg.conf). It is unknown at this time if or when this work will be undertaken, but if/when it becomes available, we may want to consider either updating this tool to allow using that interface, end-of-lifing this tool in favor of a dynamic tool, or morphing this into a hybrid tool able to set parameters either way.

Implementation

The backend (X-Kit) consists of the following:

  • A simple xorg.conf parser and writer. This could be inspired/derived from Guidance's "xorgconfig", but its API should be made easier for developer use, easier to extend, and easier to maintain.) See https://launchpad.net/xorgparser

  • An xorg.conf validator. Given an xorg.conf it should perform a sanity-check of the file. The idea is to allow xorg-conf-editing applications to detect if the user's xorg.conf is in a form that can be successfully edited, and to bail out gracefully if not.

  • An options data store. This provides the listing of available options, default values, and so on.

The frontend is a GUI dialog allowing the user to browse the xorg.conf sections that have modifiable options, to modify them, and apply the settings.

Implementation of xorg.conf Parser and Writer

XorgParser doesn't depend on any other Python module (apart from the 'sys' module) and provides a transparent way to read and write the xorg.conf. It is very easy to extend by reusing its methods which are widely documented.

XorgParser allows to get and set options, create and remove new sections, subsections, get, set and remove references to other sections, etc.

Simple API. A small set of methods should be enough to create more advanced functionalities. The main metaphor here is that each entry (e.g. driver entries, references, etc.) is an option. Only the names of the sections are hardcoded.

Easy to extend: xutils.py is an extension of xorgparser.py. It would be easy to either extend xutils or ignore xutils and extend xorgparser so as to implement more advanced features.

The code of the first release is available here.

Implementation of xorg.conf Validator

Validator should look for references to other sections inside of section. For example it would be interesting to see what references to other Device and Monitor sections are located in a Screen section (when available) or if references to Screen sections are available in the ServerLayout section.

Validator would have to make sense of the xorg.conf and see if it has a coherent structure and maybe try to fix it.

Validator should prevent the XorgOptionsEditor from making the xorg.conf unusable.

Implementation of Options Data Store

The Options Data Store is a collection of options available for each driver and the server. This data should be kept in a structured format, containing this information:

  • Option name
  • Option type: none | boolean | number | string
  • Short one sentence description

Much of this data can simply be extracted from the man pages. A script will be implemented to perform this conversion and output the XML data format. If any errors are found in the man pages from this conversion, the fixes need to be forwarded upstream for the appropriate man page. Likewise, if there is an option known to cause problems and not recommended for use, it should be removed from the man page. The idea for this tool is that the user should have access to all documented options as would one configuring things manually based on man page information.

An experimental implementation of the Options Data Store is available as of June 8 2008.

The implementation plan for the data store is:

  1. Generate at build time and include statically with the package
  2. Generate at installation time (using localized man pages present during install)
  3. Generate on the fly (using localized man pages available at run time)

For phase 1, it is permissible to modify the output of the tool to fix up problems the man page parser has with man pages. This is allowed for two reasons: First, because it may take some time to work out kinks in the man page parsing code; second, because it will take time to roll out fixes to many man pages. Having the ability to hack on the output removes risk that either of these two issues could block release. Ultimately, though, the goal is for the output of the parsers to be usable directly, with no modification; this must be achieved before proceeding to phase 2.

For phases 2 and 3, checksums of each driver's man page should be kept, so that the parsing step can be skipped and cached data used if there has been no change.

GTK GUI Frontend

Until and unless this tool is proven completely safe, the GUI application will not be available through the normal menus. Instead, the user will need to run it from the command line (much like they would regedit on Windows).

A quick mockup based on the meeting which took place at the UDS:

XorgOptionsEditor-gtk1.png

[Note: In the above mockup, two column headers are included; obviously it should list only one at most.]

UPDATE: A screenshot of the prototype (July 07 2008)

xorgconfig-07-07-2008.png

UPDATE: A screenshot of the real application (July 25 2008)

XorgOptionsEditor-25072008.png

Here it is possible to see a screencast of the prototype in action. I uses XKit's parser already.

Here it is possible to see a screencast of the real application (still WIP) in action (July 25 2008).

The GUI frontend will consist of a page showing the current xorg.conf's sections and all available options with their current settings (or their defaults if not set). A visual indicator (such as a checkbox or graying out) should be used to distinguish between options which are listed in xorg.conf vs. those that are not yet. Perhaps only active options should be listed, with an 'Add' button that pops up a secondary dialog to select options, and a Delete button rather than a checkbox for removing them?

The GUI should also permit setting the driver. This could be done either via a separate dropdown. If the xorg.conf omits the driver entirely (thus allowing the xserver to auto-select it), Xorg Options Editor will perform hardware detection and select the appropriate driver as the xserver does.

The GUI should hide all the details of the sections from users so that they will only have to click on a device in the treeView on the left and simply add, remove, enable or disable an option without having to care about which section such option belongs to. For example the user will remove "Virtual" without knowing that it belongs to the Display subsection of a Screen section.

When the user clicks on a device in the treeview, a treeview on the right will display the identifier of its respective Screen section (if available). A 'Misc' section will also be included for setting ServerFlags options (not shown in mockup).

The Description column will store only the 1st line of each description of the options. The full description might be provided in a tooltip when the user moves the mouse cursor over an option. Ideally, this should be implemented in a mechanism that won't require horizontal scrolling (see mockup).

An Apply button will back up the current xorg.conf and apply the changes when pressed. Notification should be made that the user needs to reboot.

The use of PolicyKit is a good idea since it enables us to run the application with user's privileges instead of root's.

If no xorg.conf is available, the program should automatically:

  • create a new xorg.conf
  • create a new Device, Screen and Monitor section (the Screen section should contain references to the other 2 sections) for each card
  • set a driver in the Device section by detecting the pci_id of the graphics card(s) (with lspci -n | grep 300 ) and looking up the id in /usr/share/xserver-xorg/pci/ so that, for example, a GeForce 7300 GT would be assigned the "nv" driver.

Qt GUI Frontend

As soon as the GTK frontend (XorgOptionsEditorGtk) is complete it will be easy to port it to QT4 or KDE4 since the tools used (i.e. the Parser, the Validator and the Options Data Store) are desktop agnostic.

Test Plan

  • Verify tool can set options in a variety of xorg.conf cases
    • No xorg.conf
    • Minimal xorg.conf (like was shipped in Intrepid) with no options
    • Minimal xorg.conf with the option already set
    • xorg.conf with many different devices, some of which are disabled or commented out
    • Fully fleshed out xorg.conf, such as one generated by displayconfig-gtk
    • Broken xorg.conf, with bad section layout
    • Broken xorg.conf, with options set to bad values (string vs. int, too many parameters, etc.)
    • Broken xorg.conf, with invalid/unavailable options specified
  • Verify that after setting an xorg.conf option that breaks X, the user can restore to the backed up known-good configuration, via:
    • Options editor, booted with vesa in 640x480 resolution
    • Bulletproof-X mode
    • xfix
    • Safe Mode
  • Verify that the options are displayed in the user's correct language.
    • Configure system to a different language, restart the tool, and verify the options are in that language if translated man pages are available.
    • Verify that if user is set to a non-English language that only has some of the X man pages translated, that the tool displays the translations where available, and English where not available.
  • Verify that is able to select correct xorg.conf even when hardware changes:
    • Boot with/without a wacom tablet attached
    • Boot with/without an external monitor attached
    • Install system on a removable drive, and verify the correct xorg.conf is selected when drive is used in two systems with differing graphics cards (see bug 257825)

Outstanding Issues

BoF agenda and discussion

Gobby session - UDS

https://wiki.ubuntu.com/X/OptionsEditor

 * mostly autoconfig these days, but not options for driver and for server options
 * not intended to be an end-user tool, but for power-user tweaks, testing
 * GUI on top of X-Kit (library that parses and writes xorg.conf)
 * python-xkit already used for screen resolution tool and jockey
 * initially: GTK frontend to deal with standard situations (disable Ctrl+Alt+Backspace), not universal editor
 * don't choose driver, and otherwise overlap with jockey and display configuration tool, but offer nvidia tweak options
 * ServerLayout: mostly obsoleted by xrandr, but still useful for setting up tablets; tool won't touch that for now, too dangerous
 * ServerFlags: if users report bugs, we can tell them to enable NoTrap
 
= Who is the User? (Novice vs. Expert) =
== Novice ==
 * Martin suggests making the system abstract away raw xorg concepts into more human terms ("speed up nvidia, etc...")
 * Martin suggests that perhaps it could enable multiple options at a time -> use case centric instead of option centric
 * No
 
== Expert ==
 * Bryce suggests just making it an easier way to set settings, flush, etc...
 * Bryce suggests that the tool is aimed at the user who is passionate about tweaking their X to make it performant, and will try different combinations of X settings.
 * Tools for novices should be created in a one-off manner (such as Wacom tablet configurator)
 
 * Can we add an "import option tool?" -> get new options into the tool by pulling update from the manpage
  * Yeah, if we use the man pages, it will be there, modify upstream
  * You do this at build time, not runtime (b/c) man pages won't be there
  * First line of attack will be to try to do it at run time
  
 * "Your X is broken" is not a use case
 
 * If we want to add an option, we should add it to the man page, if we want to remove an option we should remove it from the man page
 
 * Martin asks about the ability to switch to a different driver. Bryce points out that in this use case, the user is generally stuck at the command line
 
 * Gives the illusion of being an intermediate tool, but is really advanced as it adds all the options. It will invite users to "destroy" their computer.
 
= Safety Nets =
 * Rick thinks that a user has a right to expect that a GUI will not hork their system. Therefore, Rick wants this to not be available via the GUI by default. Options are to make user install it, or to make user run it from command line. 
 * Rick wants a "Factory Presets" button (to return it to the original state if their configurations) (Loïc: using a saved xorg.conf or dpkg-reconfigure -phigh ? the latter can hose both X and the console on multi-screen systems)
 * Could temporarily apply settings and then pop a dialy asking users to accept changes (ike when you change monitor resolution.
 * Should not be very visible in the desktop, since it is potentially dangerous/can break X
 * currently offers to change the driver; might be beneficial to allow user to temporarily switch to another installed and compatible driver, such as vesa, for debugging
 * Bryce: Recovery from bad options is an important part: had a "one time use only" file that rebooted, tried the new file, and if it worked, applied it as the real file.
 * xfix lets you try to auto detect and configure: could create features so that if autofix doesn't work you can try different drivers
= Backup and Change History =
 * Martin: You could always just use "one time" files and not have multiple back ups
 * Users may want to roll back to sets of changes
 * Loïc would be happy to be able to select an xorg.conf "saved" by the tool at a special time (like snapshots), which would help when moving from a 2 screen setup to a one screen setup (when 2 screen needs special configuration, i.e. separate servers for each screen, screens that aren't properly autodetected or systems with IGP and graphic cards using different drivers)
 
= GUI Design =
 * change it to be a tree view instead of a list
 * For displaying long descriptions you can have a description section under the table and then always show the description for the currently selected item there
 * There could be a "Test" button that passes the new config file to a X server on a new VT (assuming the drive is solid and doesn't destroy the X session)
 * present server flags as a top-level object, it applies to all video cards
 * show description from the manpage (it's parsed anyway)
 * Don't show "Description N/A", just wastes space; prod Bryce instead to fix the manpage
 * Apply -> Save (since you have to restart X)


CategorySpec

X/OptionsEditor (last edited 2008-12-19 16:49:37 by host65-93-dynamic)