A community-sourced user research project to answer the questions:
- What is the most intuitive and user-friendly way to extract files from an archive?
- What is the most appropriate and explanatory language for us to use when we refer to an archive?
Description
We'd like everyone to help us discover the optimum solution to the archive problems mentioned above. This is the first time we're trying user testing as a broad community activity. Please bear with us if the process is not perfect and let us know your thoughts and feedback on how the process can be adjusted or improved. Please use the Answers section of the project for your feedback.
The way we plan to find the optimum solution is to collectively conduct research to gather data, which we can then analyse and base our decisions on. This data collected will be analysed centrally by the team at Canonical. Findings and recommendations will be reported after the data has been analysed.
To ensure consistency in the way the data is collated and facilitate analysis, we have prepared the following guidelines. Please adhere to these to ensure we get valid data and find the best solution.
What are we testing?
We will be giving test participants a task that will require them to interact with an archive file on Ubuntu and talk about the experience.
Who are we testing?
It is important to select users that represent our target audience, not just our existing audience. We want to optimise the product for use by people outside the pool of Ubuntu enthusiasts.
Don't ask your family or friends who work in the computer industry or are developers. Ask family and friends in other industries, people who are willing to participate at a cafe, library, university, on the train etc. Try to get an even spread in terms of gender and age and experience with computers (but avoiding expert users).
Please conduct as many or as few tests as you have time for. If you were doing this on your own you would have to conduct at least 5 interviews to begin to identify trends, but we have the benefit of the collective effort to provide us with sufficient data to identify these and don't want to take up too much of your time.
When and where are we testing?
We would like to conduct these test during the weeks commencing 27th July and 3rd August, 2009.
Where are we testing? We're testing everywhere our users are! By conducting distributed user testing, we should collect user data from populations all around the world. More specifically, we recommend administering user tests in comfortable, populated areas like libraries, cafes, pubs, etc.
How to run the tests
Preparation
Download and set up screen capture software to record the sessions. We recommend ScreenJelly because they can capture and upload a screencast using only your browser.
- Create a new account on your computer that's unmodified – so that the interface is fresh and doesn't contain your existing files.
Log in to your fresh account and download this archive: extract_contents_to_desktop.tar.bz2
- Extract the archive on your desktop so that you have two folders called "Downloads" and "My Favorite Photos" there.
Make sure the files on your desktop (Downloads, Downloads/photos.zip, My Favorite Photos) are the same for each test.
During the test
A suggestion of how you might introduce the testing session:
"I'm conducting an exercise as part of a community trying to make technology easy to use.
"As a community we will all ask people like you to look at the product in question and ask them what they think. What’s good about it, what’s bad, what things are easy and what are hard. Then we add up all the findings from the different people and get the balanced view of where the problems are and what needs fixing. All this means I need you to do two things today:
"The first is: I need you to be completely selfish and self-centred. Just tell me what you think; whether what you see in front of you is right for you personally. You don’t need to worry about anyone else. We’ll get the big picture by talking to lots of different people.
"The second thing is: I need you to be outspoken. If you see things you like, tell me all about them. If you see things you don’t like, go ahead and tell me all about those too. I didn’t make what you’re going to see, so I won’t be hurt if you don’t like it. Basically, just try to give me a running commentary of what you’re seeing and what you expect to see and what you think of it all. The more you tell me the better.
"Now, if you get stuck on something…you’ve probably found something that makes the product difficult to use. That’s exactly the kind of stuff we are looking for. If you get stuck there will be many others who will experience the same thing. So if you do get stuck tell me about it so we can learn what the problem is and talk about how we might fix it. In the end we’ll make the whole thing much easier to use.
"The more you can tell me about what you see and what you think, the better.
"Because this is a community exercise I will be sharing the results of testing publicly... Would it be okay to record this session on video and publish the it to the web?
Please record the test using screen capture software (Screenjelly only records up to 3 minutes, so you may have to create two). Please make sure you record audio also, if possible, because it's important to hear what users are saying, not just see what they are doing.
Feel free to take notes as you speak to participants so you can provide us with additional details after the test. Sit beside your test participant rather than standing over them so it feels more like a conversation than a quiz.
Please collect the following for each test participant:
- Name
- Age
- Gender
- Profession
- Experience with a computer (from 1-10 with 10 being an expert)
Set the scene and explain the task you want them to carry out:
- "Your friend sent you some photos and you want to see them and put them somewhere you can find them later. The pictures are inside this 'Downloads' folder. Please show me how you would access the pictures and move them from there to the 'My Favorite Photos' folder." (Point to both folders on the screen)
- "Have you done something like this before?"
- "How do you think you would go about doing this?"
Ask users to begin and observe what they do and what they say. Take notes of important details but please remember we are only collecting data at this point and not analysing it.
- "Go ahead and start. Please tell me what you are doing or thinking about while you do it."
Inside the "Downloads" folder there will be an archive named "photos.zip." Inside archive there is a folder named "photos" with assorted jpeg images inside of it. To find out what they refer to the archive file as, the term do they use to describe it, ask them:
- "What is this?" (point to archive on the screen without refering to it by name)
- What is the first interaction they have with the archive (double-click, right-click or other)?
- Were they able to complete the task successfully?
- Any other interesting comments or findings
Final feedback and thank you
Finish by asking your participant to give you and further feedback they would like to on what they have seen today, then thank them again for their assistance with the project.
After the test
Please email your results and a link to your recording online to djsiegel+archivetest09@ubuntu.com. In the email please include the following information about your test participant (one participant per email please).
- Name
- Age
- Gender
- Profession
- Experience with a computer (from 1-10 with 10 being an expert)
- Had users done this (worked with archives) before? Y/N Details
- Did they know how they would go about doing it before they started? Y/N details
- What name do they use to refer to the archive:
- What is the first interaction they have with the archive (double-click, right-click or other)?
- Did they complete the task successfully?
- Other interesting comments or findings
Additional useful tips
User asks, “How am I doing?”
You’re doing fine. There are no right or wrong answers. Everything you tell me helps me to understand what would make the product easier to use.
User asks, “Is that right?” or, “How DO you do task x?”
First time they ask: I can’t really answer that, or I’ll spoil the fun. Just tell me what you think. Subsequent times: what do /you/ think?
Ask why.
If a user says something interesting or unexpected or silly, ask why. Don’t wait until later to ask why. Ask then and there. If you wait until later, you’ll need to recreate the scenario in their mind. That takes time and can be inaccurate.
Be fun.
Work hard to put the user at their ease. Make little jokes about the unnaturalness of the protocol.
Charm them.
Think of the user as a long lost, aged aunt. If you're really nice and charming, when she comes to visit for tea she might bequeath you a fortune.
Help the user to satisfy their curiosity.
If they want to know something about the interface, and it's possible in terms of time and prototype to let them try something, then do. The more you can create an atmosphere of curiosity and exploration, the better.
Don't use the words that the interface uses.
If the interface says “Enter a contact” you should say “How would you put someone’s name and address into the website?” If the interface says “proceed to checkout” you say “how would you buy all the stuff you’ve chosen?”
Encourage your participant to ‘think aloud’
If they are considering the answer to one of your questions, to talk you through the how they are making the decision (this is what is most interesting to us - *why* people get to the decision they do, not necessarily which decision they make).
Comments/Questions:
I've done the occasional bit of user testing in the past, and in the introduction to what the user should be doing, I think it is good to say something like "Please don't worry about doing stupid things or looking stupid in front of me - if you can't do the task then it means that the interface is stupid, you aren't. We need to know in exactly what ways the interface is stupid". Also, if the user asks "how are you meant to do this?", you should say "I can't tell you now", but you can follow that with "after the test I can tell you in detail, but I need to see what you do as if there is no help there first." -- mishd
We do not recommend using the word "stupid!" Don't even suggest that the user might do something stupid or be stupid, it could make them nervous! I believe the language above addresses this issue already.
gtk-recordmydesktop is a good alternative to Screenjelly. It's easy to use, produces reasonably small files (using a framerate of ~6fps seems to be good enough), has no restrictions on video duration and can record sound. It doesn't post videos on the web automatically, though. --PhilBull
Why is the name of the participant required? People are more likely to volunteer if they know that their privacy can be protected. --PhilBull
Their names are not required. Even a first name will do. Asking someone their name is a good way to get a conversation going, and it's just a common courtesy in many countries.
The "Experience with a computer" metric is subjective and difficult to calibrate. Someone who uses a specific application all day (e.g. Photoshop) may rate themselves 9/10, but might be completely inexperienced in lots of other areas. We used a few questions on the Empathy user survey to judge general competence (e.g. "Do you ever edit photos on your computer?"; "Have you ever written your own computer program?") --PhilBull
Thank you for the tip. The purpose of these numbers is to give us a general sense of the person's confidence in their abilities (they may say 10 and not be an absolute expert, but we can be sure that they are confident), and we would likely only pay attention to the number given if it is a 10 or below 3.