Gnome On-Screen Keyboard

By: Henrik Nilsen Omma

GOK is the Gnome On-Screen Keyboard. As the title implies, it is a keyboard that appears on the display as an alternative for those who are not able to use a regular keyboard. As such, it is actually a fairly complex piece of software because it has support not only for clicking on the letters with the mouse pointer, but also supports switch operation¹. It is also not just used for typing, but also caters for a more general interaction with the desktop and applications.

This report highlights some general usability issues with GOK as it appears in Ubuntu (5.10). Some of the issues highlighted here may be bugs (In which case I will file them), while others will be design features that I have not grasped the purpose of (most likely in support of hardware that I do not have). Some of the issues highlighted here will relate to the general Gnome a11y infrastructure and some may be related to the way things are set up on Ubuntu. Let's find out ...

This review is intended to be constructive criticism, and I will try to follow up with concrete suggestions for improvements where possible. It is my impression that accessibility tools often do not get the sort of thorough and critical treatment as other FOSS software does (and benefits greatly from). We absolutely need this kind of feedback if we are going to make open source platforms a serious alternative for the disabled community.

Installation and Activation

We have a known dependency bug for GOK in Ubuntu (See: 1, 2). Once you have it installed you must make sure it's enabled in the Assistive Technology Preferences window. First enable 'assistive technologies' generally and then you must enable the on-screen keyboard. This is required because GOK relies on some underlying Gnome accessibility infrastructure, but to the user it is an extra complicating step.

Ideally enabling the 'on-screen keyboard' from the Assistive Technology Preferences window should prompt you to install GOK and its dependencies automatically (and conversely, installing GOK from Synaptic or 'Add Applications' should enable it by default -- WIP).

Starting GOK

When first starting GOK, you are informed that Sticky Keys has been enabled, because GOK requires it. Why does GOK absolutely need Sticky Keys and can it not be an option?

Next, there is a pop-up window about the core pointer. I only have one pointer (mouse), so I'm not sure how I would go about configuring a primary and secondary mouse. It is clear from the GOK bugzilla that the whole core pointer issue is a complex one, due to the way X allows several pointers to be configured and the tendency for some applications to grab the pointer focus, something that will be a show stopper for someone who uses GOK for overall computer navigation. Apparently it is advised to test GOK using two mice, which I would expect reduces the amount of testing it gets. It also excludes those use cases where someone actually wants to use GOK with a single pointer device, possibly for good reason (bug filed here).

The information window itself is exceptionally unhelpful. It suggests you go to 'Help' for more information, but 'Help' doesn't contain any more information about core pointers than you can glean from the preferences window or the above info window. Also, how is 'OK' different from 'Cancel'? Does it make a difference which I press? While the core pointer issue is technically complex, it should still be presented to the end-user in a more tidy way.

Finally, there is the information window telling you that Assistive Technology Support is not enabled. It appears that you can continue without enabling assitive technologies (clicking 'Continue'), but GOK itself an 'assistive technology' isn't it? What aspects of GOK am I missing out on? I suspect these are user-group specific, so if I need them they are absolutely vital and if I don't need them I never will (so don't show me this pop-up every time). I suspect the answer here is to have a more comprehensive setup guide that takes the user through a series of understandable questions that relate more to different use cases than to technical back-ends.

If I do want to enable 'Assistive Technology Support' ('Enable and Log Out'), is it really necessary to force me to log out? Yes, I know this is because Gnome needs to start an underlying toolkit, but surely it shouldn't require me to log out. Will GOK appear at the login prompt now, so I can log in again independently²?

The GOK interface

Having navigated the error screens you get a fairly simple main menu window, with the option to start the Composer (keyboard) and some other control functions which allow you to manipulate windows and the mouse cursor in scanning mode (I won't be reviewing those at this time as I don't have the relevant hardware).

Composer

The composer initially comes up in something resembling a QWERTY layout. It's not quite QWERTY, because the keys are perfectly lined up above each other, and not slightly staggered like a QWERTY keyboard. A large number of keys, including function keys are included, though it's actually not the standard 101 key layout either. There are several alternative layouts, including one where the letters are ordered alphabetically, and one ordered by letter-use frequency. The latter sounds like a good idea, and it probably is a good idea in scanning mode, where this would reduce the time required to get to the desired letter, on average. In pointer mode though this layout seems quite confusing, esp. as numbers are mixed in with letters.

It is possible to design your own keyboard layout for GOK, although it is not obvious how this is done or where they are stored. I attempted to load one from the keyboard preferences window, but this crashed the application. I will explore this feature further though, try making some custom layouts and report bugs as appropriate.

I think the typing layout should have an option where only the most commonly commonly used keys (letters, shift, comma and period) were visible by default and the rest were available in additional panels that would slide out or pop up when needed (see suggestion section).

Sticky Keys

I would think it should be possible for GOK to track it's own shift states, without using Sticky Keys. Using Sticky Keys can cause problems for VNC sessions at the moment, though I'm not sure whether this is a VNC bug or a Sticky Keys bug (filed here). It also appears that the Sticky Keys integration is not perfect in GOK, because it happened several times that pressing the shift key on the virtual keyboard produced capitals on the keyboard, but not in the application I was typing into (I will investigate further and file a bug).

Look and feel

And then there is the general aesthetics: GOK does look a bit clunky and garish, especially in Composer mode. Does this matter much? Well, it does if you are going to be using a piece of software all day. It would be preferable if it looked nice and integrated well with the rest of the user interface. I'm not saying the project maintainers should make this a high priority, but perhaps some artists can step in and help out?

You can choose a mode where the keyboard borrows its look from the native theme, but at least in my case, this turned out to be very bland, and the keys became difficult to tell apart at a glance. It would be preferable to have a setting with some use of colour to aide in identifying the keys, but slightly less garish than the original.

It would also be nice to have it integrated better with the desktop environment. It is possible to 'dock' the keyboard, but this does not work optimally either, as it covers up the task bar and really does take up a lot of space. It would be cool to have an icon on the task bar that could be clicked to slide out the keyboard.

Windows and Mac

In Windows XP, when you start the on-screen keyboard program, a small QWERTY-like keyboard appears. When you click on the keys, you get letters in your editor. That's it. Obviously this is a much simpler application that is not by far as configurable. But it does actually support switch operation and sets an example for simplicity.

Surprisingly, OSX, which seems to lead the field in accessibility support generally, does not have an on-screen keyboard included by default. I do not have a Mac, but this was my clear impression from their website. Instead you have to purchase a third-party application for this purpose. Doing a review of all the third-party on-screen keyboard available for Windows and Mac is clearly beyond the scope of this article, but these are obviously part of the software landscape for those who need one.

The good points

• It is very flexible (though I haven't figured out how to use all these features)
• Users with quite severe disabilities can use it, as long as they are able to operate a single switch. This is very important because it potentially open up a mode of communication to to people who might otherwise have none at all.
• It's free! -- This is also important, because even though it provides a fairly simple function in its basic form, the proprietary alternatives can be very expensive.

Suggested changes

Setup wizard

This piece of software tries to cater to groups with very different needs. That's a good aim, but it seems that there is a need to customise it further for each group within that framework. Once you start making compromises and settling on a best average over several user groups you risk limiting the usefulness of the tool severely for some groups. Specifically, it seems to be intended to work mainly with multiple input devices at present, and core pointer mode (just one mouse) seems to be somewhat broken. Again, I think the answer is to have some sort of setup wizard allowing the user to specify the relevant hardware configuration.

Improved keyboard layout

The default full 101-key keyboard is not well optimised for typing with the mouse or for switch-scanning. It would be better to have the most commonly used keys available by default (the letters) and have the other keys be available through expansion panels. I've made a mock-up picture of what that might look like. This would allow you to have larger letter keys which are easier to see and aim for and/or have the keyboard use less screen real estate.

In pointer mode, I suggest a keyboard layout based on QWERTY (or Dvorak, or a local variety, or whatever you like) that uses colour to distinguish the most commonly used letters. That should make it easier to hit these keys faster. I've made a mock-up of a possible colour coding scheme, but I'm sure there are better patterns we could come up with. Following this letter frequency table I've marked those in the > 5% frequency group as red, the 5% > f > 2% group as blue and finally < 2% group as green. I have no research to indicate that this grouping is especially sensible, but it does provide a distinctive pattern that makes it easier to quickly locate a given letter, once you get used to the colour pattern.

Users

Does GOK have any users? Seriously. Or does it only ever get tested by it's developers? It certainly has many potential users. Those who currently have to pay large sums for a full-featured on-screen keyboard on Mac or Windows. I would actually recommend making Windows and Mac ports. This would see GOK being used by real users, providing useful feedback, not to mention an invaluable free service to those users.

It is a concern that accessibility for Linux is such a niche topic that it doesn't receive enough testing and development. This seems to be especially true in the case of GOK. Gnopernicus, Dasher and GOK are mentioned as answers to the questions of whether we have AT features on Linux, but do these products really meet the needs of users? At the moment I fear it's more of an alibi.

What about tablet PCs? Providing a convenient way for traveling executives to enter text on their tablet PCs is obviously not the core remit of the GOK project, and due to the core pointer issue, this currently works poorly. This need will only grow, however, and eventually someone is going to come along and write a Free on-screen keyboard that caters to a general tablet PC audience. This will most likely either be as a fork of GOK or completely from scratch. If that happens it will be a great opportunity missed, because if GOK became the standard application for tablet-PC input on Linux, it would get much more testing and development resources that it does currently. Not only would GOK improve, but it would be tested for compatibility against the rest of the desktop to ensure that all applications played nicely with it. We might even then see accessibility support on the desktop enabled by default, removing the need for all those pop-up windows mentioned in the introduction.

Comments

Actually, OSX does have a built-in keyboard viewer, and it has a very simple and pleasant layout. There's an article about it here: http://www.apple.com/macosx/tips/keyboard11.html -- DavidGoodger

It is hard to know how many GOK users there are. Over the years I've been emailed personally by individual gok users, and by a teacher who required gok for a class of 15-20 students. There has also been curb-cut interest from kiosk folks as well as other on-screen stuff. I hope gok can help anybody who requires it and welcome this critical review. Thanks Henrik. -- David Bolter