A simple on-screen keyboard

• Launchpad Entry: https://launchpad.net/distros/ubuntu/+spec/compiz-mag

• Created: 2006-06-02 by HenrikOmma

• Contributors: HenrikOmma, SvenJaborek

• Packages affected:

• Status: BRAINDUMP

Summary

A new screen magnifier for low vision users using compiz-based rendering technology.

Rationale

The current magnifier gnome-mag is slow and resource instensive. compiz holds the promise for much better magnification rendering via hardware accelleration.

Use cases

Reading a news website

Take this news page: http://news.bbc.co.uk/1/hi/uk/4752819.stm It has some text and pictures and a great deal of non-essential navigation items (most other newspapers are worse). For the sake of example, let's consider a user who might use 2x zoom to get a general overview of a desktop of webpage layout but would need 8x zoom on normal text to read it.

It would be useful to be able to zoom in on the different elements separately. Imagine that you could select the picture first. Press some hotkey to skip to the picture, and have the picture by itself displayed at 8x zoom where you can pan around. Another keystroke would give you the caption overlayed at the bottom of the picture in a high contrast colour or it could be read out via the screen reader.

You then want to move on to the main body text. You return to the 2x overview page. You press another hot key to highlight some text. The first block of text appears with a heavy bright red (customisable of course) frame around it or with a full rectangle at 50% opacity so you can pick out the block you want. the next hotkey or mouse command would then zoom that text up to 8x, filling most of the screen. It would be wrapped to fit nicely and could be scrolled if needed. It would be as if you selected the text, copied and pasted it into a special app that would magnify it to your liking or read it out if you preferred. You could also select your own fonts.

Document composition

In using a word processor like OpenOffice you can set the font size very large or the zoom to a high factor in OpenOffice itself, which might be sufficient to read the text. However there are several problems:

• The text entry cursor is narrow and difficult to spot
• The toolbar buttons, slider widgets, etc. are small and in low

contrast grey tones

• The menubar menus and context menus are small

You could go for a large high-contrast theme, but in fact you might prefer to keep the buttons and widgets small so they don't take up too much space and distract from the text you are working on.

Solution: You should be able to hover over the toolbar and have the buttons zoomed by 8x and the contrast enhanced. You move the mouse over to the right and the scrolling widget grows a large handle in high contrast that you can easily grab and scroll. The context menus also appear zoomed.

Scope

The must-have features

• Fixed zoom factors - This could be pre-set by the user, but it should also be possible to switch zoom levels on the fly with a hotkey
• Ability to follow the text-entry cursor - This requires AT-SPI magic, but is a vital feature.
• Ability to jump to window that gains focus, e.g. after clicking "New Mail" - also requires AT-SPI.
• Positioning mouse in the center of the screen via one keypress in case it gets lost
• Configurability - As many settings as possible should be configurable by the user (but it can be a simple config file to start with). Possible options are (not all must-have!):
  • Colour and size of the mouse pointer
  • Mouse guides (e.g. circle, cross,...)
  • Speed of motion / Sensibility

  • Position of mouse -> e.g. freely movable versus fixed in center

  • Fixing a certain area on the screen, e.g. task bar or panel

Some desirable features

• Variable zoom factor - Fixed factors may not be exactly what you want. It would be cool to be able to adjust the zoom level precisely with the mouse wheel, say.
• Scriptability - Again Orca has shown that being able to tailor the behaviour of the AT app to your working application can be very useful. This might well apply to the magnifier as well.
• Fit window to zoom area - If you are working in Gedit and you have zoomed in to a comfortable level, you now want to make sure that the whole window fits into the viewable area. You then press a hot key and Gedit's window is resized so that it fits.
• Interaction with screen readers and braille systems - This adds another level of complexity but could potentially be extremely powerful. You could browse at a moderate zoom level and then have sections of text read out to you.
• Colour schemes that can be switched on the fly - this is more flexible than the system's fixed colour schemes. Simple invert mechanisms may possibly be adopted from other screen magnifiers?!

Design

We are going to extend the zoom plugin, which exists in compiz. It works like a magnifier glass, where the glass is on fullscreen size. The performance of the zoom plugin is impressive and moving is very smoothly without high CPU usage like on gnopernicus. Another possible solution is the xmcm project (sourceforge), which works with X extensions like composite. At the moment the plugin system in compiz bashes us with some constraints at the keyboard-input front. For a proper implemention we'll have to access (and perhaps modify) the keyboard input and input redirection is not yet implemented in the xgl server.

The communication interface should work through dbus. Compiz is going to implement dbus this in the future, because other plugins need to be remoted too.

The cursor tracking will work on at-spi much like it does in gnopernicus. The cursor tracking will be a standalone module, communicating with the core via the interface.

Implementation

Core

Compiz is created in the c language, same for the xmcm code.

The interface to the core will need to transfer these information:

• magnifier_glass_position(x,y,s) (x,y: coordinates of top-left pixel, possible values in desktop size range. s: screen, possible value number of virual desktops ).
• magnification_scale(int x)

Its usefull to transfer commands like to control the core, then others can build a gui for it.

• start_magnification(settings)
• stop_magnification()
• move_to_window (windowid)
• fit_window_into_glass (windowid)

Cursor Tracker

t.b.d.

Script Engine

Both use cases need a fair bit of information from the application via AT-SPI and there needs to be custom script knowledge about the application and individual user preferences. In the case of the BBC website the system would also need to know where the useful information usually is located.

The script engine can be programmed in every language, since it only needs to communicate to the core.

Outstanding issues

Input-redirection in xgl is needed for a proper compiz plugin. Build problems with xmcm on newer systems like dapper and fc5.

BoF agenda and discussion

References