Gathering statistics on user interface choices

We need hard statistics on what people do with their computer. Physical usability testing has its limitations, it's costly, time consuming and does not cover a wide range of people. Currently, this is the only way we can gather data to backup our decisions with adding, removing or changing user interface components on the desktop.

Why this idea rocks hard:

  • Even if a community member doesn't know a lot about heuristic evaluation or human-computer interaction, they can opt-in to the program and send data to the design team and therefore feel like they are helping improve Ubuntu design.
  • These hard statistics will provide solid reasoning for design team decisions, giving some rational to what seems like crazy, pot smoking behavior. If you remove a button, and people start complaining, you can turn around and say "well, thanks to this testing program, we collected data from 100,000 users over 3 months and found they only used this button 19% of the time."
  • Publicizing the data to both our community and upstream will improve our relationship with everyone. We have the largest amount of users of any Linux distribution, and a fair amount of them are willing to contribute but don't know how. What better way than to submit useful information in the background to help make Ubuntu easier to use?
  • GTK already draws the windows, the buttons and the menus. We'd just need to add a few lines of code that collects the amount of times a user hits some object (or doesn't) and then submit that data every so often. Of course it would be totally optional, similar to how popcon works now, except we'd improve the backend so it doesn't suck balls.

What do we get?

  • Better applications, better interface, better user experience. Happier users.
  • Improvement of collaboration between the community and the design team.
  • An easy way for people to collaborate to Ubuntu UX and design, and make a difference. (Not just posting meaningless crap to Ayatana and getting ignored).
  • Far, far better reasoning for design decisions.
  • A happier community overall.

Kevin's Spiel

There are often discussions on the Ayatana mailing list (and elsewhere) about ideas for improving usability and other user interface elements.

Typically, these conversations start with a proposal or complaint about something that's recently changed. The next few emails offer lists of the obvious pros, cons, and other considerations.

Following that, the thread devolves into bike-shedding and a myriad of me-toos and counterexamples.

Unfortunately, basing design decisions on this anecdotal evidence doesn't lead to the best design experience for the majority of users. Instead of relying on anecdotal evidence, we should instead make design decisions based on data collected from a wide variety of users.

Bringing people into the lab to test our software has a few problems: it's expensive, time-consuming, and doesn't allow for long-term use studies.

I propose that we establish a project that allows people to opt in, passively collect about their usage and habits, and submit the data to a central location for aggregation and analysis.

We would create instrumented versions of some libraries (such as Gtk) and applications that can record data as the participant uses the software on her own computer in her home/work environment.

As we discuss new user interface designs and we have questions about how users are currently using an application or feature, we can update the framework to collect the appropriate data. When the participant downloads the latest version of the framework, it will start collecting the data we're interested in. After the data has been collected, the participant can view the data and choose whether or not to submit it. The submitted data is aggregated and used to answer the questions posed.

Currently, Mozilla is running a similar project with Firefox called Test Pilot ( Test Pilot is an add-on for Firefox.

usabilty-testing-gtk (last edited 2010-05-10 12:50:48 by godbyk)