Ubuntu Wiki

You can help improve Ubuntu by testing the usability of different parts of Ubuntu.

Usability is the "Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use."

Usability testing consists of bringing a person with little or no experience with the product you want to test and the product itself together, and ask the person to use the product to complete certain tasks or achieve certain goals. As the facilitator of the test, you record how the tester reacts to the product, and how she tries to achieve the goals that you put forward. This data can be very useful to show where usability problems occur.

Traditionally, Usability testing involves multiple cameras to record not only what happens onscreen but also how the users use their hands and how facial expressions change as they attempt to complete the various tasks. This is quite extensive, and generates a lot of data - both quantitative (number of successes vs. failures at each task) and qualitative (how the users react, their emotions and stress at the given situation), as well as general behavioural data - which may be dependent on the users' level of experience with similar products.

Basic Usability Testing

You won't need any equipment to do basic usability testing, just observe your tester and take ample notes. Here's a few pointers on how to get started:

Before the test

• Get friends who are not used to Ubuntu or the Ubuntu website to be testers. If you invite people you don't know well to be testers or even offer them some reward for helping out with testing, make it very clear that any reward is not at all contingent on their performance and that we know there are problems and are doing the testing in order to find them.
• Do your best to make your testers feel comfortable when you introduce them to the usability test: Offer them something to drink and small-talk like you would with any guest you might have over.
• Clearly define the task and tell the test person that you want to see how easy or difficult the task is.
• Ask if there's questions about the goals. Once testing begins, the facilitator doesn't answer questions (instead referring to the sheet of paper).
• Make it clear beforehand that you do not want them to feel that they're embarrassing themselves, and that you cannot help them if they get stuck. You want them to do their best on their own - and they won't do that if they know you're standing by to help them.
• For usability testing, you want as much feedback as possible. Ask the testers to talk about every step. Have them think aloud: Their rationalizations, frustrations and successes. What did they expect to happen? What happened? Why do they think it happened?
• Be cautious with asking too many questions during the test, better explain what you want them to do in detail before you start the testing, and ask follow-up questions afterwards.

During the test

• Just watch the tester as they go about completing the task you've set them. Don't help! Just watch!
• Observe each task and note:
  • How long it takes,
  • how many clicks it takes
  • how often people are successful
  • how confident the users are as they do it.
• Make notes of where people get stuck. If they seem to spend a lot of time on a particular page or have to bounce back and forth a couple times, make note of this.
• Don't wait to take notes. Do it during and/or immediately after.
• Also make note of the type of computer OS they're use in their everyday life. We may find that Windows XP users have no troubles but Mac users struggle.

After the test

• Organize your observations. There are two different types of observations: Subjective and Objective. Subjective observations are, "it seems like..." while objective are "it took 8 clicks..." Clearly note when you are expressing an opinion and as much data as possible that lead you to the opinion. For example, "I noticed that users clicked back to the XYZ page several times before finding the correct link to PDQ. Therefore I think we need to make PDQ higher on the page." The subjective part is that PDQ needs to be higher. The objective part is that people had to click back to the page several times.
• Send your data to the ubuntu-desktop mailing list, or another forum where the appropriate developers will see it.

Advanced Usability Testing

If you have access to recording equipment, you can easily put your own usability lab together. A suggested setup would include:

• A computer for the tester to use
• Another computer to record the audio and the VNC session (the on-screen action)
• A Microphone
• A facilitator (you) to facilitate the testing who makes the tester feel comfortable

and introduces the test.

• Goals and relevant details about the tasks to be tested (such as login info if necessary) provided on a piece of paper.
• Use [www.realvnc.com realvnc] to record the screen of the computer
• Use a microphone or telephone recording setup to record their voice and encourage them to talk about their experience as they do it (sync the video and audio for later by having them say click with their first click)

Current Usability Testing Projects

The following is a list of areas where we would like more active usability testing. Follow the link for more information on procedure and who to contact for further details.

Ubuntu Installation

• https://launchpad.net/products/ubuntu-website/+spec/usability-help

Ubuntu Website

• https://launchpad.net/products/ubuntu-website/+spec/usability-downloadiso

• https://launchpad.net/products/ubuntu-website/+spec/usability-shipit

Ubuntu Desktop

...

Use Scenarios

It may be useful to tell the tester a simple background story, thus giving a context for performing the tasks that you want to test which the user otherwise would not try out.

There are various ways to do this:

Simple

"Let's pretend your boss has asked you to get a copy of Ubuntu, u-b-u-n-t-u, working on your computer some time in the next week. How would you accomplish this?"

This is open-ended and easy. But can lead to frustration and stress if the tester feels that her boss will take note of any bad performance that she might deliver in the test.

Verbose

"You're a gourmet chef, besides running a successful restaurant in New York, you have also written several cookbooks to great critical acclaim, and you have now been approached by a television network to do a series on how to do good cooking easy. Preparing for the TV-series, you've been given an office to use at the TV station, complete with a desktop PC. Leisurely, you slink into the office one Thursday afternoon to do a little research on the Internet. You learn back in the comfortable office chair and sip from a long-stemmed glass of well-aged Bordeaux while the computer boots up. The desktop appears to be called Ubuntu."

It may well prove important to avoid all kinds of unpleasantness and stressful situations in the background story, because the user will feel unnecessarily stressed and will consequently perform worse. For instance, if the is told tester to get some information from one of their co-workers' computer and that co-worker happened to run Linux, it may make the tester stressed and angry - not at the computer or the Linux system, but at the imaginary co-worker for indirectly forcing her to deal with something she wasn't familiar with.

If you get people relaxed and in a good mood, they're much more likely to give good (as in qualitatively useful) feedback on the elements you want them to test.

Other resources

To see current Usability concerns, please read: UsabilityCaseStudies UsabilityWishlist

To learn more about Usability Theory, please read: Usability UsabilityTheories WhyUsability

For more inspiration on how to do your own usability testing, see Novell's http://www.betterdesktop.org website.