UDS Maverick

These are the notes from the session during UDS Maverick about touch support.

Rolling touch wiki page: X/Blueprints/Touchscreen

What can we do for Maverick?

• Simple modifications to apps to be more touch friendly
  • gtk touch screen mode will tweak appropriate gtk widgets to make them more touch friendly
  • bigger icons?
  • other themeing changes?
• extend compiz to support gestures

Testing

• Use bug tag 'touch'
• Make cursor theme that is a big circle (to emulate visible region when using finger)

Post-Maverick

• on-screen keyboard
  • want xkb + ibus support, discoverable, be able to change orientation (for tablet/slate)
  • onboard: only xkb, only English layout
  • need someone to make one for xkb + ibus
  • probably will take two cycles (i.e. not Maverick)

Integration options

• toolkits
  • Qt (has full support already)
  • gtk+ (some support upstream)
    • scrollbars (flick-to-scroll) are needed. There is a kinetic scroll widget that someone made (for Midori).
    • can't distinguish between finger and mouse (to change how UI responds). perhaps set a 'mode' based on whether screen is rotated
• onscreen-keyboard
  • should be hooked up to entry-widgets (Qt, gtk+)
  • should be overlayed on UI (taps get recognized, swipes "fall through" to windows under onscreen-keyboard)

apps to target

• compiz (wm) could get gesture support
• app indicators/windicators?
• web browser
  • chromium is becoming touch friendly. getting rid of menus
• media player
  • banshee created whole new UI: kubano (sp?). It uses no menus, very big widgets (like big album art images on a flickable view)
  • rhythmbox: album/artist browser is small
• document viewer (not creator)
• empathy
  • some bits like the expanders for groups are too small
• shotwell
  • good except small fonts/targets on sidebar (sidebars are common problem -- can we create a generic sidebar widget and modify apps to inherit from that)

Unity

• app indicators, menus will need to collapse
• need to regenerate fonts for subpixel rendering when rotated (do after delay to allow UI to be responsive immediately)
• app indicator and menus may be spaced too closely
• Spec for touch is being worked on (don't have right-click)

Guidelines

• Anything menu-driven or near the edge of the screen will be difficult to use
• Stylus is very different than finger
• Bigger buttons
• default spacing/sizing changes for menus so they globally get more touch friendly when appropriate
• tablet/netbook screens tend to be wide aspect ratio because that's what cheap for manufacturers

touch onscreen keyboard requirements

• create one from scratch [arnegoetje can mentor, but not code]
• needs to be tied with keyboard indicator to change layouts on demand
• needs to work with IBus to change layout whenever a different IME is chosen
• get layouts from XKB
• for IMEs we need to produce layouts by ourself
• store the layouts in the ibus packages (modules)
• add function keys, like arrow keys, etc. (configurable)
• detach layout from the actual keys submitted to IBus (i.e. display Bopomofo, but submit Latin characters from US English layout)
• use python-virtkey for emulating modifier keys.
• use SVGs for the keys being displayed
• accessibility
  • when user moves his finger over the key, use text-to-speech to tell the user which letter is under his finger
    • needs text-to-speech framework [luke]
    • needs hints for the text-to-speech synthesizer to tell the user which letter he is dealing with
    • use espeak?

Actions

• [ACTION] File bugs for touch related issues with applications and desktop (using tag 'touch')
• [ACTION] investigate gtk touch mode
• [ACTION] make a cursor theme with large cursor that simulates finger such that apps can be tested
• [ACTION] find someone interested in making an onscreen keyboard

Comments

[bryce 2010-05-07] (Actually, these days the X side of things is pretty minimal; X's evdev driver is basically just a thin wrapper around kernel level support. Most of the remaining X bits like -evtouch and tslib are becoming obsolescent in favor of evdev.)

[rickspencer3 2010-05-11] What about calibration? http://github.com/tias/xinput_calibrator/tree/debian

[brunogirin 2010-05-15] Some observations that you may want to take into account.

On #1, what about introducing MPX in order to support multi-touch? Widget toolkits and applications need to make use of it for it to be useful but AFAIK Qt4.6 will support it and having support at the X level is the first step to get multi-touch enabled apps.

Integration options: to change how the UI responds, the simplest way may be to detect hardware capabilities, such as:

• is there a touch-based interface?
• does the touch interface support multi-touch?
• is there a pointing device plugged in (mouse, trackball, graphic tablet, etc)? -- note that this can change dynamically when unplugging a USB mouse for instance
• is there a physical keyboard plugged in? -- this can also change dynamically

and then decide what profile to use based on capabilities:

• touch only => use a touch optimised interface with soft keyboard

• pointing device => use a standard interface, even if the device is touch-enabled

• no physical keyboard => enable soft keyboard

What about supporting devices like the Nintendo DS that has 2 screens, one of them touch-enabled, the other one a standard display? In practice, this may be exposed as a display with half its area touch enabled. Would this impact this blueprint?

Guidelines: there is no concept of "hover" with a touch based interface. You only know the user's finger's position when the user actually touches the screen which means that the position of the cursor is only available after a "mouse down" event. This means that the accessibility requirement for the onscreen keyboard above can't work: you only know what key the user's finger is over *after* he presses that key. The only exception to that rule are pressure sensitive touch screens (light pressure = hover; heavy pressure = touch) so maybe that would be another hardware capability to take into account.

For comprehensive feedback on using Ubuntu on a touch-based device, some users in the ubuntu-uk mailing list have been experimenting with the O2 Joggler and have created an EtherPad here: http://pad.ubuntu-uk.org/Joggler