UDS-M

UDS Maverick

These are the notes from the session during UDS Maverick about touch support.

Rolling touch wiki page: X/Blueprints/Touchscreen

What can we do for Maverick?

  • Simple modifications to apps to be more touch friendly
    • gtk touch screen mode will tweak appropriate gtk widgets to make them more touch friendly
    • bigger icons?
    • other themeing changes?
  • extend compiz to support gestures

Testing

  • Use bug tag 'touch'
  • Make cursor theme that is a big circle (to emulate visible region when using finger)

Post-Maverick

  • on-screen keyboard
    • want xkb + ibus support, discoverable, be able to change orientation (for tablet/slate)
    • onboard: only xkb, only English layout
    • need someone to make one for xkb + ibus
    • probably will take two cycles (i.e. not Maverick)

Integration options

  • toolkits
    • Qt (has full support already)
    • gtk+ (some support upstream)
      • scrollbars (flick-to-scroll) are needed. There is a kinetic scroll widget that someone made (for Midori).
      • can't distinguish between finger and mouse (to change how UI responds). perhaps set a 'mode' based on whether screen is rotated
  • onscreen-keyboard
    • should be hooked up to entry-widgets (Qt, gtk+)
    • should be overlayed on UI (taps get recognized, swipes "fall through" to windows under onscreen-keyboard)

apps to target

  • compiz (wm) could get gesture support
  • app indicators/windicators?
  • web browser
    • chromium is becoming touch friendly. getting rid of menus
  • media player
    • banshee created whole new UI: kubano (sp?). It uses no menus, very big widgets (like big album art images on a flickable view)
    • rhythmbox: album/artist browser is small
  • document viewer (not creator)
  • empathy
    • some bits like the expanders for groups are too small
  • shotwell
    • good except small fonts/targets on sidebar (sidebars are common problem -- can we create a generic sidebar widget and modify apps to inherit from that)

Unity

  • app indicators, menus will need to collapse
  • need to regenerate fonts for subpixel rendering when rotated (do after delay to allow UI to be responsive immediately)
  • app indicator and menus may be spaced too closely
  • Spec for touch is being worked on (don't have right-click)

Guidelines

  • Anything menu-driven or near the edge of the screen will be difficult to use
  • Stylus is very different than finger
  • Bigger buttons
  • default spacing/sizing changes for menus so they globally get more touch friendly when appropriate
  • tablet/netbook screens tend to be wide aspect ratio because that's what cheap for manufacturers

touch onscreen keyboard requirements

  • create one from scratch [arnegoetje can mentor, but not code]
  • needs to be tied with keyboard indicator to change layouts on demand
  • needs to work with IBus to change layout whenever a different IME is chosen
  • get layouts from XKB
  • for IMEs we need to produce layouts by ourself
  • store the layouts in the ibus packages (modules)
  • add function keys, like arrow keys, etc. (configurable)
  • detach layout from the actual keys submitted to IBus (i.e. display Bopomofo, but submit Latin characters from US English layout)
  • use python-virtkey for emulating modifier keys.
  • use SVGs for the keys being displayed
  • accessibility
    • when user moves his finger over the key, use text-to-speech to tell the user which letter is under his finger
      • needs text-to-speech framework [luke]
      • needs hints for the text-to-speech synthesizer to tell the user which letter he is dealing with
      • use espeak?

Actions

  • [ACTION] File bugs for touch related issues with applications and desktop (using tag 'touch')
  • [ACTION] investigate gtk touch mode
  • [ACTION] make a cursor theme with large cursor that simulates finger such that apps can be tested
  • [ACTION] find someone interested in making an onscreen keyboard

Comments

[bryce 2010-05-07] (Actually, these days the X side of things is pretty minimal; X's evdev driver is basically just a thin wrapper around kernel level support. Most of the remaining X bits like -evtouch and tslib are becoming obsolescent in favor of evdev.)

[rickspencer3 2010-05-11] What about calibration? http://github.com/tias/xinput_calibrator/tree/debian

[brunogirin 2010-05-15] Some observations that you may want to take into account.

On #1, what about introducing MPX in order to support multi-touch? Widget toolkits and applications need to make use of it for it to be useful but AFAIK Qt4.6 will support it and having support at the X level is the first step to get multi-touch enabled apps.

Integration options: to change how the UI responds, the simplest way may be to detect hardware capabilities, such as:

  • is there a touch-based interface?
  • does the touch interface support multi-touch?
  • is there a pointing device plugged in (mouse, trackball, graphic tablet, etc)? -- note that this can change dynamically when unplugging a USB mouse for instance
  • is there a physical keyboard plugged in? -- this can also change dynamically

and then decide what profile to use based on capabilities:

  • touch only => use a touch optimised interface with soft keyboard

  • pointing device => use a standard interface, even if the device is touch-enabled

  • no physical keyboard => enable soft keyboard

What about supporting devices like the Nintendo DS that has 2 screens, one of them touch-enabled, the other one a standard display? In practice, this may be exposed as a display with half its area touch enabled. Would this impact this blueprint?

Guidelines: there is no concept of "hover" with a touch based interface. You only know the user's finger's position when the user actually touches the screen which means that the position of the cursor is only available after a "mouse down" event. This means that the accessibility requirement for the onscreen keyboard above can't work: you only know what key the user's finger is over *after* he presses that key. The only exception to that rule are pressure sensitive touch screens (light pressure = hover; heavy pressure = touch) so maybe that would be another hardware capability to take into account.

For comprehensive feedback on using Ubuntu on a touch-based device, some users in the ubuntu-uk mailing list have been experimenting with the O2 Joggler and have created an EtherPad here: http://pad.ubuntu-uk.org/Joggler

X/Blueprints/Touchscreen/UDS-M (last edited 2010-05-20 18:21:58 by eth3)