Chromium/utouch Integration

Introduction

This page describes the integration of utouch-frame and utouch-grail with Chromium to enable sophisticated gesture recognition within Chromium (and Chrome). Both the architectural and technical aspects of the patch (appended to this page) are explored. The summary concludes with a list of open issues/work items.

Chromium Architecture

Chromium is a cross-platform web browser built on top of WebKit and serves as a basis for Google Chrome. Moreover, it serves as the basis of ChromeOS. From an architectural point of view, the overall system can be divided up into two components:

• The UI component

• The WebKit component

In the UI component, different platforms and/or GUI toolkits are abstracted in terms of so called modules. Within every module, platform specific bits and pieces are defined and implemented in files that are named in a platform specific manner (e.g., by appending _x11 to the filename).

Crossing the boundary between the UI component and the components glueing WebKit functionality to Chromium-specific bits and pieces is realized by an IPC mechanism.

MultiTouch & Gesture Recognition Architecture

Both WebKit and WebKit Chromium define interfaces and types related to processing input from multi-touch devices and carrying out gesture recognition tasks. Only recently, these interfaces and types have been moved to the UI component of Chromium, more specifically to the Aura subcomponent. Aura is a GUI Toolkit abstraction that works on top of OpenGL ES 2.0 and will eventually become the basis for implementing both Chromium's and ChromeOS's user interface across all major platforms.

In Aura, handling of multi-touch devices is carried out by the singleton TouchFactory (ui/base/touch/touch_factory.h). Currently, its implementation (ui/base/touch/touch_factory.cc) relies on X11 and XI2.x. In general, the class takes care of:

• Maintaining a list of currently connected direct-touch devices
• Setting up event processing for these devices

The patch accompanying this document extends the functionality of the class by associating a utouch-frame instance with every instance of the class. Every XI2.x event is passed through utouch-frame (relying on its default X11/XI2.1 backend). Moreover, the utouch-frame instance is exposed to other classes for gesture recognition purposes. Finally, we altered the class TouchFactory to provide support for dependent touch devices like the Apple Magic Trackpad.

Gesture recognition tasks are abstracted by the interface GestureRecognizer (ui/aura/gestures/GestureRecognizer.h). Our modifications introduce an implementation of this interface in terms of utouch-grail and provide recognition of:

• Two-finger drag gestures
• Two-finger pinch gestures
• One- and two-finger tap gestures

both on direct and dependent touch devices (thanks to the modifications to TouchFactory). Every instance of an implementation of the interface GestureRecognizer is associated with an Aura RootWindow (ui/aura/root_window.h and ui/aura/root_window_host_linux.cc) and the recognizer instance is only invoked for touch events that "belong" to the respective root window. That is, we dispatch hit-testing and window identification tasks to Chromium in general and Aura in particular. Whenever the grail instance recognizes a gesture, we provide the surrounding framework with an instance of class aura::GestureEvent (see ui/aura/event.h). Please note that we modified the class to support pinch-gesture specific parameters, i.e., providing the current scale factor in x and y direction.

Please find a video demonstrating pixel perfect scrolling within Chromium Aura here. The screen cast has been recorded on the following setup:

• Precise Alpha
• utouch-frame and utouch-grail trunk versions
• Apple magic trackpad

And here is an overview of the architecture:

with a more utouch-centric point of view presented below:

Summary & Open Issues

The patch accompanying this document introduces gesture recognition functionality to Chromium Aura based on utouch-frame and utouch-grail. To this end, we introduce a utouch-frame instance relying on its default X11 backend to the class ui::TouchFactory and provide an implementation of the interface aura::GestureRecognizer (see ui/aura/gestures/gesture_recognizer.h) in terms of utouch-grail (see ui/aura/gestures/gesture_recognizer_grail.h and .cc).

In the future, we might want to abstract the multi-touch event processing even more and move the utouch-frame instance away from class ui::TouchFactory to our implementation of interface aura::GestureRecognizer. To this end, we would need to provide a Chromium-specific backend to utouch-frame and alter the class aura::TouchEvent to provide all relevant information to utouch-frame. Nevertheless, by abstracting touch-event handling on top of chromium, we do not need to take care of platform specific issues in utouch-frame but dispatch the platform-specific tasks to Chromium.

Finally, a list of work items to implement the more abstracted approach presented just before:

1. Provide an abstraction of touch device addition/removal events (status: mostly done)

2. Modify class aura::TouchEvent to provide all required information (status: wip)

3. Provide a Chromium backend for utouch-frame (status: not started)

4. Move utouch-frame instance and event processing to grail-specific implementation of aura::GestureRecognizer.