Wikimedia Research/Usability testing

This page describes the basics of how, when, and why to do usability testing.

What usability testing is good for

 * Figuring out whether users can complete important tasks, like finding an article, uploading an image, or adding a citation
 * Comparing whether different versions of the same feature, like Visual Editor or Wikitext editor, work better for a certain type of user.
 * Learning what users expect to happen when they perform a new task for the first time.

What usability testing is not good for

 * Learning users' opinions about why they would use your software.
 * Learning users' opinions about whether they would choose your software vs. your competitors.
 * Learning what users think would make your software better.

Define a test plan
Create a written summary of what you why you want to perform a usability test, and how the results will be used. This will help you decide which tasks to include in your test (Create a protocol) and which kinds of users to test with, and how many to recruit (Recruit participants).

Create a protocol
A protocol is a list of tasks that test participants will perform with your software. To determine which tasks to include, start out by writing a list of questions that you want to answer with the study.

The protocol should contain basic instructions about the task they are being asked to perform. The goal is to give the user enough information that they understand the nature of the task, and where to start, but not to tell them how to accomplish it. These instructions should be written out, so that all users get the same amount of explanation.

Some advice for writing task instructions
 * Use plain language and short sentences. Avoid using technical jargon ("diff", "revert", "wikilink"), unless you are 100% sure your users understand exactly what those terms mean.
 * Make the goal of the task clear. Tell the user what you want them to accomplish, but don't tell them how to do it.

Recruit participants
You should recruit participants for your test who have similar goals, needs, and characteristics as the people who use your software. All software is used by people with diverse backgrounds and levels of experience, and particular tasks or features will be more important to some users than others. Generally you will want to test the usability of your software for a particular type of user.

In order to understand your users better, consider developing a pre-test questionnaire and asking your users to fill it out before they begin their first task (or ask them verbally, and write down their answers. See Take notes.)

Facilitate usability tests

 * Make it clear to the user that you are not testing them, their technical skills, or their mind-reading abilities. In usability testing, the users are the experts: they are helping you test the software, so you can make it better.
 * Lead them to that starting point. Don't make users do unfamiliar things that are outside the scope of the tasks you've assigned them. If you are testing whether people can change their preferences, help them log in first. If you are testing an image upload workflow, direct them to a page with an "Upload an image" link.
 * Remind them to think aloud. Ask the user to describe what they are thinking, what they are trying to do, and what they expect to happen, right now. You will probably have to remind them repeatedly: that's expected. Make sure to remind them gently.
 * Don't lead them through the task. The goal of a usability test is to understand whether users can complete tasks on their own, under "normal" circumstances. When a user struggles with a task, it can be tempting to jump in and tell them what to do. Don't give in to this temptation.
 * note: if the participant wanders so far afield that they are no longer performing the task you assigned them, feel free to direct them back to the starting point (in a polite and supportive way). Similarly, if a participant can't complete the task at all after multiple attempts, and has completely given up in frustration, you can give them hints to get them back on task.
 * Ask questions. In most usability testing sessions, the most valuabel findings come from asking questions in the moment. If a participant seems confused, surprised, delighted, frustrated, etc by something they see, or something that happens, ask them to explain what they are thinking/feeling and why. It can also be useful to ask users what they expect will happen when they do something, right before they do it. Then you can follow up with them after they see the results of the action (clicking a link, entering a search term) by asking "So what happened? Is this what you expected? Why or why not?"

Take notes

 * have a designated note-taker.
 * record quotes.
 * audio or video record the session.
 * record the time for important notes.

Analyze results

 * discuss results with other observers right after the test.
 * type up/review/consolidate your notes.
 * note common problems across users.

Report results

 * prioritize based on severity.
 * prioritize based on design goals.
 * share quotes and clips.
 * provide suggestions/next steps.