Introduction

The data cross walk method is designed to collect usability data to answer pre-defined usability questions. The method is prized for its ability to:

1. Distribute data collection across observers, while maintaining some degree of consistency in results
2. Integrate observers who are not trained UX professionals into the process
3. Quickly produce both qualitative and quantitative results.

At the heart of the method is a crosswalk document that defines:

1. The usability questions being considered.
2. A situation that will result in observing something that will answer the question.
3. The instructions to read to the user.
4. The specific data to capture.
5. An empty column to record the results.

This document is reproduced for each participant that is observed. All of these documents can then be easily compiled, and a report generated.

Example of Creating a Crosswalk

Step 1: Define Usability Questions

Let's assume that we want to investigate Add/Remove Applications. The first step is to think about the questions that we have regarding the feature area. Here's a list that we might come up with:

1. If a new users wants to add a new program that is in main, but was installed by default, will they find Add/Remove...?
2. Can new users find an application to match a certain function even if they do not know the specific application for that function?
3. Can new users find and install an application that is not in main if they know the name of the application?

We can probably come up with a slew more questions, but for this example, three is probably a good number.

Step 2: Set up Situations

Now we can think up some situations which would result in users doing something worth observing relevant to our question.

We can imagine that if we set up situations like this, we will be able to observe users trying to achieve tasks relevant to our questions.

Step 3: Create User Prompts

For the sake of consistency in observations, it is quite useful to have the user read something that tells them what to do. This has the advantage of having all participants in the study responding to the exact same thing (though some users will require and ask for clarification from). So we'll add a column to capture what exactly we will tell the user.

Step 4: Define Data to Record

Now we'll define what data we want observers to record. Note that the more data you ask for, the less consistency you will get. However, the less data you ask for, the less richness you'll get. We'll also go ahead and add a blank column for collecting the data.

Running Participants

Preparing a computer

You need to ensure that you have a computer that is ready for participants to use, and that you can easily reset to the base state for new participants. Many people find that setting up a virtual machine and taking a snapshot of it in the correct state makes this much easier.

Setting up the task list

Simply copy the user prompts from the third column in a new document. It's usually best to print out each prompt on it's own page so that user do not read ahead.

Preparing the participant

It is important each participant has a minimum of performance anxiety. Many participants will feel uncomfortable trying things that are probably hard to do for the first time, while being observed. For this reason, it is important to tell the participant that you are testing the software, you are not testing them. They are helping you test the software. This may need repeating if the participant shows signs of stress. You should also be prepared for the opposite situation. Some participants are quite prolific in their criticisms, and it's important not to get defensive in these situation.

It is also quite helpful if the participant can think aloud while they are working. This will give you some insight into what they are thinking, and makes it easier for you to follow along. Before the session starts, ask the to tell you "everything they are trying to do, how they are trying to do it, and what happens after they try". Participants often become quiet when they are concentrating. If the participant starts to work without thinking aload, a simple "hmmm?" will often prompt them to start thinking aloud. Otherwise, "please remember to think aloud" also helps.

Recording the session

A video record of the session can be very useful. Istanbul or Gtk Record My Desktop are good options for this. Strive to capture the user thinking aloud, as that will make the video twice as useful.

Capturing data

Focus on filling in the cross walk, as that will be compiled into the report. However, it may also be useful to keep a separate set of more detailed notes to engender more discussion about the sessions.

More

• One of the best books for getting started with usability is Don't Make Me Think. Note that the book is mostly about web usability, but the section on testing is relevant to all software.

• Read more about think aloud here and here