Touchscreen

Summary

Identifies packaging, development, and testing tasks which would improve touchpad support on Ubuntu.

Rationale

Many touchscreens fail to work out of the box on Ubuntu and require excessive amounts of work to set up. This situation should be remedied so touchscreens are simpler to get working. While in the past touchscreens were a specialty type of hardware, these days with netbooks and other small form factor devices, touchscreens are becoming more prevalent and are more likely to be seen as targets for Ubuntu.

Scope

Scope covers finger-operated screens. Does not cover touchpads or tablets. multi-touch is out of scope for the time being.

Primary focus is low level kernel/X support along with some configuration tool work. Assumes kernel support, and does not cover application-level support aside from testing.

The amount of work required to establish general support for all manner of devices is rather overwhelming and likely beyond what can be achieved in the scope of one blueprint's worth of work. Thus we presuppose that there exists specific target hardware platforms that efforts will be focused around.

Design

Hardware/Driver Support

Touchscreen hardware is supported by a diverse set of libraries, calibration tools, and configuration approaches. The good news is that there is a single standard approach now; the bad news is that little hardware is currently supported by this system. See http://tias.ulyssis.org/calibration/ for a table showing what software to use with which hardware.

Thus, there are two ways to do touchscreen support: The easy way, and the right way.

The easy way involves identifying one of the many drivers that works with the given hardware, and shim it into Ubuntu by crafting a customized xorg.conf (or other config), duct taping the relevant calibration tool into place, and verifying that the desired applications are available. This might be okay if we care only about a specific device and have hardly any manpower or schedule available to do things the right way, but would really be a waste of effort over the long term.

The right way is to ensure the touchscreen device is recognized by the kernel and can be driven using the X -evdev driver. We ensure all the underlying functionality is exposed and configurable via Xinput2. Hardware-specific configuration is implemented using udev rules (see /lib/udev/rules.d). Ideally, the user should not be involved in any configuration work aside from calibration.

It may be that even with -evdev working down in the kernel, a higher level X driver is still required. This could be -usbtouchscreen, -evtouch, -tslib, or so on. Ideally, the functionality these secondary drivers provide needs merged into the more common codebases.

The lack of a decent calibration tool is one of the main problems as currently users are configure the touchscreen in xorg.conf. A mockup of a new calibration tool is shown in JauntyTouchscreenHandling. This will need to be developed.

There is a proof-of-concept xinput daemon called x-bus which provides a backend service for handling Xinput2 on-the-fly configuration changes. This needs to be ported to ordinary C so it can be included on the LiveCD.

Window Manager UI Functionality

For a smooth user experience with touchscreens, the window manager will need updated to support several features:

  • Display rotation using multi-finger drag gesture
  • Zoom using a "pinch" gesture
  • Larger gnome panel
  • Scale up icons to a more finger friendly size
  • Fatter scroll bars
  • Larger window resize handles (or force all apps to be full screen or fixed pos/size)
  • Smart identification of "what you meant to hit", ala iphone

On-screen Keyboard

There are several different options for on-screen keyboards, including CellWriter and matchbox. CellWriter is probably closest to being ready for prime time.

CellWriter is in universe. The keys are small, requiring a mouse or stylus to operate so this will require updating to permit finger operation so it is more touch-friendly. It will also need integration into the window manager so it can be called up as needed. It will also need integrated into the gtk+ toolkit so it works with all the common widgets (GtkEntry, GtkCombobox). Possibly the onscreen keyboard could be triggered via iBus.

Application Support

Applications worth testing the touchscreen functionality against include:

  • *xournal
  • world of Goo

  • numpty physics http://numptyphysics.garage.maemo.org/

  • chess/checkers/solitaire
  • mypaint.intilinux.com
  • firefox grab and drag plugin (not packaged)
  • fuller scree (?)
  • fennec (?)

Implementation

Hardware Support

  • Identify one or more target touchscreen hardware platforms to focus on [tseliot]
    • Purchase and expense one touchscreen device [tseliot]
  • Migrate all touchscreen hal rules to udev [pitti]
  • Identify common touchscreen devices not currently supported by -evdev
    • Does kernel recognize device?
    • Does kernel evdev provide support for all buttons, input properties, configuration settings, etc.
    • Are all kernel evdev attributes exposed in the X -evdev driver?
    • If another driver provides more complete support, evaluate porting that functionality to evdev

  • Implement support for rotating screen and touch interface together (Bug 217182)

Window Manager UI

  • Investigate existing patches to Metacity or other window managers for touchscreen support
  • Implement display rotation using multi-finger drag gesture
  • Implement zoom using a "pinch" gesture
  • Implement larger gnome panel
  • Scale up icons to a more finger friendly size
  • Implement fatter scroll bars
  • Implement larger window resize handles (or force all apps to be full screen or fixed pos/size)
  • Evaluate work for smart identification of "what you meant to hit", ala iphone

On-Screen Keyboard

  • Investigate any existing work done to morph CellWriter into a finger-driven keyboard interface

  • Investigate any existing work with other onscreen keyboards

  • File a MIR for CellWriter

  • Review CellWriter bug reports and plan a triaging/upstreaming program

  • Enlarge keyboard keys to a more finger-friendly size
  • Implement patch that integrates cellwriter into Gtk so can be easily launched when entering data into any text widget

  • Enable easy triggering of CellWriter via iBus or other appropriate protocol

Testing

  • Document test cases on X wiki (bryce)

  • Create test cases for appropriate applications
    • World of Goo
    • Chess
    • Solitaire
    • Mypaint
    • Firefox with grab and drag plugin

References

Outstanding Issues

BoF agenda and discussion

CategorySpec

X/Blueprints/Touchscreen (last edited 2010-07-12 07:31:42 by 210-242-151-101)