Original spec by frafu with extended design details and implications.
Summary
Add additional configuration to the mouse behaviour in X and/or Gnome to make the desktop more accessible for users of alternative pointers and those with difficulty using a normally configured mouse. This spec describes five enhancements:
left-click-hold context menu: left click&hold to open the contextual menu
dwell-clicking: perform a left click, a left double click, a right click or a left click&drag without pressing any mousebutton.
tremor-suppresing: Some users are not able to keep the mouse completely motionless and would welcome an option to smooth out mouse movements.
mouse-gestures: triggering of specific actions by doing gestures with the mouse
Rationale
The standard functions of the pointer (usually the mouse) has not much changed over the years. It is used to point, to select, to click&drag and to open the contextual menu. However some of this functionality is not available to users of special adaptive pointers and others (like the elderly) can find the mouse difficult to use. Some simple (optional) tweaks to the mouse driver functionality is needed to fix this. Some of these features will also be useful for 'average' users and power users.
Use cases
left-click-hold context menu:
James is a disabled user that is only able to use the left mouse button. When he wants to open the contextual menu (which is usually opened with a right click), he has to do it in an indirect way: for example by selecting switch button in onboard. If the contextual menu appeared with a left click&hold, he would have an efficient and direct way to do it.
Cindy uses a tablet pc, that she controls mainly with a pen. The left click&hold would enable her to open the contextual menu with the pen.
Apple PowerMac (PPC) traditionally have a single-button mouse. You can access the context menu with a modifier key, but the click&hold is more intuitive for many.
dwell-clicking:
- Michael is a disabled user that uses an eyetracking device to move the pointer. However the device does not provide a way to do the clicks. Combined with dwell clicking, he gets the same functionality of a normal 2 button mouse.
tremor-suppresing:
- Albert is a 73 year old Ubuntu user. Because of tremours in his hands he finds it difficult to select. As his hand has gotten tremblier with the age, he would appreciate if the system could ignore part of the erratic movements that he does with the mouse.
mouse-gestures:
- Anna is a poweruser that recently switched from Windows to Ubuntu. She was accustomed to use the mouse gestures software in Windows and would be happy to find an equivalent on Ubuntu.
- Jim is a disabled user that is able to move the mouse but unable to click. By using the mouse gestures software defined below in timed mode, he will not only be able to do the various clicks, but he also has many commands available by merely doing mouse gestures.
Scope
Add functionality to the xorg mouse driver and implement a configuration panel in Gnome.
Design
MouseTweak #1:
- This mousetweak opens the contextual menu of the item that lies under the pointer, when the user keeps the first mousebutton pressed for a determined amount of time without moving the pointer.
In order to avoid conflicts with the click&drag, the system could proceed like this: if after the mousedown, the mouse does not move for n milliseconds (for example specified with a slider), open the contextual menu; if the mouse moves before the specified time has elapsed, go for a drag&drop.
Of course, the current n has to be greater than the current double-click time; for simplicity reason, one could also generally make: minimum(n) > maximum (double-click time), as I did in my example tab below.
- The time n of opening the contextual menu 1.5 seconds after the mousedown is probably a good default value.
- The contextual menu stays opened as long as the users keeps the left mousebutton pressed.
- When the user releases the left mousebutton, the command of the contextual menu under the pointer is executed and the contextual menu is closed. If the pointer is not over the contextual menu when the button is released, the contextual menu is closed without anything else happening.
The left click&hold for the contextual menu should be optional. A check box in the mouse control panel and/or in the accessibility control panel should allow the user to enable and disable the function. Its default shipping state should be disabled.
- There is also a checkbox to automatically enable this function at login. (We could drop the 'Automatically enable...' checkbox and make the next session use the same settings of the previous session.) If available, this checkbox should be disabled by default.
Below you can see a sample preference window built with 'Glade Interface Builder' that would offer a left click&hold; it is simply the 'Buttons tab' of the Mouse Preferences shipped with Ubuntu Edgy to which I have added (among others) a frame for the left click&hold.
MouseTweak #2:
- Dwell clicking would belong to the accessibility features of the mouse.
- See below for a sample configuration dialogue for dwell clicking:
- The checkbox 'Enable Dwelling' is used to enable the dwell clicking for the account of the current user. Its default state should be disabled.
- The checkbox 'Automatically enable after login' will automatically enable dwell clicking for that user after he logs in. Its default state should be disabled.
- The user can choose between 2 ways of doing the dwell clicking (radio buttons in the configuration dialogue):
Dwell clicking with the help of a window to choose between left click, right click, double click and drag&drop. (Dwelling with ClickType Window). This is the default dwelling type when active.
- When dwelling is enabled, and this way of dwelling is selected, the 'Select Click Type' window appears on screen.
- The system clicks on the pointers location, if the pointer stays without moving on that location for the time set in 'Time to wait before click' (let us call this time the dwell delay). A dwell delay of about 1.5 seconds is probably a good default value.
- The type of click which is performed corresponds to the click type that is set in the 'Select Click Type' Window.
- To change click type, the user has to move the pointer over the button that corresponds to the click type that he wants, and wait for the dwell delay to elapse without moving. The system then activates the button, regardless of what click type is currently set.
- If the checkbox 'Automatically switch back to single click mode' is activated, the dwell clicking function automatically reenables the left click after another click type has been performed. If this checkbox is not activated, the current click type does only change when the user selects another click type in the 'Select Click Type' window. The checkbox is activated by default.
- The 'Drag And Drop' click type works like this (This is also used for example to select text):
- the user moves the pointer over the item to drag
- after the dwell delay has elapsed, the item is grabbed (mouse button down)
- the user sees the grab because the item gets selected or better: we could make the pointer change its shape
- after the grabbing, the system watches for the pointer to be move for another dwell delay. Two situations are possible:
the pointer does not move before the second dwell delay elapses: the drag&drop did merely correspond to a long left click. The drag&drop terminates when the second dwell delay has elapsed. (Textmark A)
the pointer moves before the second 2 dwell delay elapses: depending on the context: an item is dragged, text is selected,... The drag&drop terminates only after the pointer has been motionless for another dwell delay. (In other words: if during the dragging, the pointer stops shortly and restarts moving before the dwell delay has elapsed, the dragging continues.)
- Detailed description of how to use the contextual menu when dwell clicking:
The user chooses the right click type in the ClickType Window
- Then he moves the pointer over the item whose contextual menu he wants to open
- After the pointer has been motionless over the item for the dwell delay, the contextual menu opens.
- If I am correct, the contextual menu in gnome remains open until the next click; we don't need to change this when dwell clicking. We have only to be aware that after the click to open the contextual menu, we have to move the pointer to have a second click performed. The second click occurs after the pointer has again been motionless for a dwell delay.
- Depending of whether the checkbox 'Automatically switch back to single left click mode' is enabled or not, the click after opening the contextual menu is a left click if the checkbox is enabled or a right click if the checkbox is disabled. As Ubuntu allows both clicks to choose a contextual menu item (or am I wrong?), it works regardless of the setting of that checkbox. It is also possible that the click after the opening of the contextual menu is performed outside the contextual menu. In this case, I don't see any reason for the system not to behave as if it was a left/right (depending on the checkbox) click of a normal mouse.
- Dwell clicking where the type of click is determined by time and a gesture:
- When this type of dwelling is used, the pointer cycles between 2 states: the pointer state and the dwell state; the user can determine the state of the pointer from its shape: in pointer state we see the usual arrow and in dwelling we see the dwell shape.
- The cycling only occurs when the pointer is motionless.
- The dwell state appears after the pointer has been motionless for the time determined by the "delay before dweelling starts" slider. A good default value might be 1.5 seconds.
- The dwell state remains for the time determined by the "duration of mouse/dwell state" slider. After that time has elapsed, the pointer changes back to pointer state for the time determined by the "duration of mouse/dwell state" slider. After another delay of the time determined by the "duration of mouse/dwell state" slider has elapsed, the pointer switches back to dwell state... And so on as long as the pointer remains motionless. A good default value might be 1.5 seconds. For simplicity reasons, we could drop the second slider and set "delay before dweelling starts" equal to "duration of mouse/dwell state".
- If the pointer begins to move when it is in its pointer shape, it is a normal mouse movement.
- If the pointer begins to move when it is in the dwell shape, the system checks in what direction the movement began:
- If the movement started in the left direction, perform a left click on the coordinates of the screen where the pointer was motionless
- If the movement started downwards, perform a left double click on the coordinates of the screen where the pointer was motionless
- If the movement starts to the right, perform a right click on the coordinates of the screen where the pointer was motionless; there might be an item with a contextual menu at these coordinates (as the usual behaviour of the contextual menu is to remain open until the next click, it is also possible to work with contextual menu when dwell clicking with gestures)
If the movement started upwards, it is a drag&drop (which corresponds to a mouse button down):
In order to avoid conflict with pointer shapes depending on the context (for example caret in text selection), it might be wise for the pointer to drop its dwell shape as soon as the drag&drop has started. (Or add a checkbox to the dwelling configuration dialogue and let the user choose whether he wants the pointer to keep its dwell shape until the end of the drag&drop.)
- Depending on the context, grab the item under the coordinates where the pointer was motionless and drag it around, or begin selecting text, or...
To terminate the drag&drop (which in fact is the emulation of the mouse button up), the pointer has to stay motionless for a determined delay (which could again correspond to "delay before dweelling starts". (put the dragged item where the pointer stopped moving, or end the selection where the pointer stopped moving, or...)
After performing the left click, left double click, right click and drag&drop, the pointer returns to its pointer state and the cycling begins again with the "delay before dwelling starts" time.
MouseTweak #3:
MouseTweak #4:
Above you can see a sample GUI for the mouse gestures functionality of the pointer. There are 3 ways for doing mouse gestures:
- by clicking a mouse button and optionally holding a modifier
The user can choose between a left button single click&hold, a left button double click&hold, a right button click&hold and a middle button click&hold.
The "Click normally if there is no movement in:" setting is especially useful to distinguish a click&drag from the mouse gestures.
- If the gesture takes more time than the timeout allows, the gesture is ignored.
- by indicating the gesturing with a key
- Of course, in this mode, the user should use a key that he does not need in normal usage. A Function key or a modifier could be good candidates
- by starting the gesture in determined time intervals.
- When gesturing in this mode, the pointer has 2 different states:
- Pointerstate: the pointer behaves as a normal mouse when it is moved and clicked.
- Gesturestate: if the pointer is moved while in gesturestate, the movement is interpreted as a gesture.
- The state of the pointer at the start of the movement determines whether it is a mouse gesture or not.
- When the pointer is motionless, it toggles continually between pointerstate and gesturestate.
- The pointer keeps the state it had at the start of the movement as long as it is moved.
- The length of the pointerstate+gesturestate can be defined by the cycle parameter.
- There is also a parameter available for the user to define how much time of the cycle the pointer has to be in pointerstate; the rest of the time of the cycle the pointer is in gesturestate. The cycle begins in pointerstate.
- At the end of a movement (regardless of it being a gesture or not), a new cycle begins. There is one exception: if the action of a gesture is "toggle visibility of the pointer", the pointer remains in gesturestate as long as the pointer is invisible.
- The user is able to distinguish visually whether the pointer is in pointerstate or gesturestate; for example by changing its shape or colour.
- The "Change the colour of the path when the gesture is not recognized" checkbox is greyed when the "Draw the path of gesture" checkbox is disabled.
- When gesturing in this mode, the pointer has 2 different states:
When clicking on the "Define Gestures..." button, the user gets to the dialogue where he can define the mouse gestures for all and for specific applications:
- On the left in the Mouse Gestures Definition Dialogue the user can add and remove applications for which he wants to define gestures.
- The Global and Dialogue entries of the list cannot be removed.
- The gestures defined in global work in all the applications that are in the list.
- If the checkbotton "Enable global gestures in unlisted applications" is enabled, the gestures defined in global also work in the unlisted applications.
- To disable gestures in an application, the user has to add it to the list and enable the "Disable all gestures in this application" checkbox.
- If the user defines a gesture in an application and the same gesture is also defined in global, the gesture definition in the application overrides that in global; in other words, the action defined in the application is executed and the action defined in global is ignored.
- When the "Disable global gestures for this application." checkbox is enabled, only the gestures defined for that application trigger an action.
- The "Disable global gestures for this application." and the "Disable all gestures in this application" are not visible when the user selects Global or Dialogue in the list of applications.
- In the Dialogue entry in the list of applications, the user can define gestures that are available when the front window is a dialogue. These gestures override the gestures defined in Global and in the application.
- On the right in the Mouse Gestures Definition Dialogue the user can see the list of gestures that he has defined for the application he has selected in the list of applications.
- When a gesture is performed by the user, the system first checks whether that gesture is defined in the list of gestures for the active application. If a match is found, the corresponding action is executed. If there is no match, the search goes on in the list of gestures defined in global and the action defined in global is executed if a match is found here. Is there is no match in the active application nor in global, nothing is done except for the changing of the colour of the path if the corresponding checkboxes are enabled.
- He can add, edit and remove gestures for the selected application by clicking on the appropriate button in the Gestures Definition Dialogue.
- By clicking on the button to add or edit gestures, the following dialogue appears:
- In the first field, the user can give a name to the gesture.
- With the left, down, right and up buttons, the user defines the movements of the gesture.
- With the popup, the user defines the action that is executed by that gesture. The popup provides the following choices:
- Some actions require a parameter, others not. Here is a more detailed description:
- gda: Do Nothing: no parameter
- gda: Ignore Next Gesture: no parameter
- gda: System Volume Up: no parameter
- gda: System Volume Down: no parameter
- gda: Toggle Mute: no parameter
- gda: Close Window: two radiobuttons: first: close the window where gesture occurred; second: close the front window
- gda: Minimise Window: two radiobuttons: first: minimise the window where gesture occurred; second: minimise the front window
- gda: Zoom Window: two radiobuttons: first: zoom the window where gesture occurred; second: zoom the front window
- ga: Next Window: no parameter
- ga: Previous Window: no parameter
- ga: Quit Application: two radiobuttons: first: quit the application where the gesture occurred; second: quit the front application
- ga: Next Application: no parameter
- ga: Previous Application: no parameter
ga: Execute MenuItem: text entry field (single line) to enter name of menu item to execute
- gda: Execute Keystroke: text entry field (single line) to enter keystroke to execute
- gda: Type Text: text entry field (multiple lines) to enter text to type
- ga: Open Selectiom: no parameter
- gda: Open App/Doc: text entry field to enter name of app/doc and a button that opens the choose file dialogue to select the app/doc
- gda: Open URL: field to enter url
- d: Press OK: no parameter
- d: Press Cancel: no parameter
- d: Press Apply: no parameter
- gda: Left Click: no parameter
- gda: Left Double Click: no parameter
- gda: Right Click: no parameter
- gda: Drag and Drop: no parameter
- gda: Toggle Pointer Visibility: no parameter
- Remark:
- the popup menu in the Gesture Editing Dialogue has different entries for the global/application gestures and the dialogue gestures.
- gda means: g=entry available in the global popup menu, d=entry available in the dialogue popup menu, a=entry available in the application popup menu
- to have a detailed description about how the click actions work, please refer to the dwelling with gestures specification of mousetweak#2.
Implementation
I don't know whether it can be useful, but in the following zip package you can find the files that I created with glade to present the sample gui:
mousetweaks.gladeversion2.12.1.zip
MouseTweak #1:
* Under MacOSX there are at least 2 utilities (1. One Finger Snap, 2. Look Mom No Hands,...) to open the contextual menu with a left click&hold. Both are not free.
Code
MouseTweak #2:
* gok provides dwelling functionality, but only locally in the gok application; not systemwide. Could it be a starting point?
* kmousetool is an utility that does a left click whenever the mouse stops moving; right click has been introduced through a gesture, but with a major drawback: it is not possible to do a right click without the left click as it always does a left click when the mouse stops. I tried to contact the author about this but did not get any reply. I think that the original MouseTool software is now shareware. On the other hand, there is also a version of kmousetool that is part of the kdeaccessibility software.
Code
MouseTweak #3:
Code
MouseTweak #4:
The following may provide some starting points for the implementation:
- In the Universe Repository, there is the mouse gesture software wayv that according to the comments in Ubuntu's Forum does not work well enough to be useful:
[ForumThread http://ubuntuforums.org/showthread.php?p=2012107]
When looking in Settings->Input in the Configuration Window of KDE, there are entries about mouse gestures. So KDE seems to be shipped with mouse gestures. I think the program that provides this functionality in KDE is called kHotKeys.
- kmousetool (see implementation of mousetweak #2)
Code
Unresolved issues
MouseTweak #1:
- Is the contextual menu needed at the login screen?
If yes, how to enable left click&hold to open the contextual menu at login time?
If no, should it not be nevertheless working at login screen for consistency reason? Or does in this case the fact prevail that a left click&hold is not a standard gui gesture and therefore disable it at the login screen. In any case, it should be implemented so to also work at login.
The best way to resolve this is probably in combination with the accessible login feature. Here is the url of the corresponding spec: https://blueprints.launchpad.net/distros/ubuntu/+spec/access-gdm
MouseTweak #2:
At the login screen, there should also be a way to activate dwell clicking; I think again that this question is better suited in the specification about an accessible login screen: https://blueprints.launchpad.net/distros/ubuntu/+spec/access-gdm
- What is better: a checkbox to automatically enable dwelling after login or making the account keep its activated/deactivated state across logins.
MouseTweak #3:
- Personally, I don't know how the tremor reduction works. Does it have to differentiate between smoothing a moving mouse and a motionless mouse. I assumed the former situation in my sample Mouse Preferences Tremor Tab above. But it is merely an assumption.
MouseTweak #4:
- Maybe we should drop the Dialogue entry in the list of applications for simplicity reasons and simply add the corresponding actions to global. In this case however, the user has to pay attention not to override in a specific application a gesture that he defined for dialogues in global.
BoF agenda and discussion
MouseTweak #1:
MouseTweak #2:
MouseTweak #3:
MouseTweak #4:
Comments
MouseTweak #1:
The activation of the left click&hold to open the contextual menu conflicts with dwell clicking with a window to choose click type. (See Textmark A in the design section of dwell clicking for the point of conflict). So the user should be told about the conflict so to have the opportunity to choose which to activate.
- Depending on the timings set in mousetweak#1 and mousetweak#4,there might be a conflict. See first comment for mousetweak#4.
MouseTweak #2:
- The Human Interface Guidelines (HIG) 2.0 of Gnome also define copy and transfer actions for the middle mousebutton in combination with modifier-keys. I don't think there is a point to make dwelling emulate the middle mouse button for several reasons:
- The dwell user can do the copy and transfer actions by using the cut copy paste commands of the Edit menu or the contextual menu.
- I don't think that a middle mouse button added to the dwelling with the help of a window is more effective than using the Edit menu, because the click that is necessary to choose the middle click type can be used to open the Edit (contextual) menu.
- When dwelling with gestures, the addition of the middle button click could enhance the effectivness, as we could define a stroke for these actions. But it would come at the expense of simplicity: At the moment, the is a click type for the 4 direction of movement (left, down, right, up). To introduce another button click we would have to add oblique directions (for example northeast) or introduce strokes made up of 2 direction executed one after the other (for example a move down followed by a move left).
The HIG 2.0 forbids to attach an action to the middle button that cannot be done in another way. Here is the corresponding quote from the HIG 2.0:If you do intend to use the middle button for a different purpose somewhere, only do so as a shortcut for experienced users, and only for operations that can also be performed without using the right button or middle button. Consequently, the dwell user probably also has a way to do every action as he can emulate the left click, left double click, right click and drag&drop.
The activation of dwelling with a window to select clicktype should deactivate the left click&hold to open the contextual menu because there is a conflict between them. In fact doing a drag&drop without moving the mouse between the two dwell delays could be interpreted as a left click&hold, because it is in fact an emulation of a long mouse button down. (See Textmark A in the design section of dwell clicking). Anyhow, it does not make sense for a dwell user to activate mousetweak#1, because normally he is not able to use a mouse button, and because the dwelling software provides its own way to emulate a right button click.
- Mouse Gestures in timed mode can be configured to provide dwell clicking with gestures and more. (see mousetweak #4)
MouseTweak #3:
* mikecorn says on the #ubuntu-devel-discuss:
- "Mouse drivers need more precise control of very small movements"
- "The method: when mouse speed slows down to a few pixels / second, increase the ratio of mouse steps to pixel steps. Normally this is 1:1, but it could become 2:1 or 3:1 as the speed goes down. This is the reverse of the acceleration table built into mouse drivers, whereby the ratio goes in the other direction as the mouse speed increases."
- This might be related to tremor reduction tweak
MouseTweak #4:
- The mouse gestures software has to make sure that the time set in "Show contextual menu after" of mousetweak#1 is greater than the time set in "Click normally if there is no movement in:" of mousetweak#4. Otherwise the contextual menu opens while the mouse gesture software is still waiting for a mouse gesture.
- Mouse Gestures in timed mode can be configured to provide similar functions (even more functions) as those defined in dwelling with gestures in mousetweak #2.