Usability Testing

In Brief

Usability testing involves observing people trying to complete a series of tasks while using an interactive product. The product can be any level of fidelity, from a paper mockup to a fully functioning product. Users are asked to perform tasks, and the observer records whether the user is able to complete them, as well as their level of confusion or frustration during the process.

Helps Answer

How do people use the product or service?
Do people understand how to use the product or service as intended?
What do people experience at different touchpoints while using the product or service?

Tags

Qualitative
Value proposition
Customers
Description

Time Commitment and Resources

Usability tests with 5 users can be finished in half a working day with minimal resources. Tests typically require no more than 5-7 users, unless the tasks are complex and involve several parties collaborating simultaneously to complete the test. Tests are often performed with extensive equipment, including full usability labs with cameras, eye-tracking software, and one-way mirrors (although this is not strictly necessary).

How to

Prepare:

Usability tests can be performed in any environment. In some cases, a usability lab with no distractions may be very different from the environment where the product will actually be used. A real environment such as a workspace or coffee shop may provide more accurate real-world behavior, but may make detailed observations difficult or impossible. A usability lab allows for detailed observations but might lack realism. Preparation is critical for usability testing since the test produces a number of potential biases in interpreting the qualitative results. At the minimum, the experimenter should have:
- An introduction script.
- A list of use cases and tasks.
- Context of the task (“You are thinking of buying a car.”).
- A description of task ("You would like to compare prices of various cars.”).
- A reasonable time limit for each task.
- Recording equipment such as a notebook or a camera.
Frame for feedback:
- The experimenter explains the purpose of the experiment to the user to ensure that they are willing to give honest, open feedback. Normally, the experimenter will reassure the user that any tasks that are difficult are not the fault of the user, and that any problems encountered are exactly the feedback the experimenter desires.
- This is normally done with a prewritten script to ensure consistency between sessions and experimenters.
Explain the task:
- The experimenter explains a single task and any context to be performed by the user.
Observe the user:
- The experimenter observes the user attempting the task while asking them to talk out loud about their impressions, intentions, and expectations. The experimenter does not explain or provide any guidance, but only interjects to ask the user about their thought process, feelings, or experiences.
- The entire process can be recorded with audio, visual, and screen-capture software, or even eye-tracking software for additional review later.
Repeat if necessary:
- The user may then be given additional tasks, and steps 3-4 are repeated until all tasks are completed.
Exit Interview:
- Usability tests are usually concluded by thanking the user and asking open-ended follow-up questions to clarify their experience.

Interpreting Results

Even usability experts sometimes don’t agree on the interpretation of usability tests. Having multiple observers for the usability tests can help eliminate potential subjective biases from having only one experimenter.
The experimenter and any observers must synthesize their observation notes, taking care to note any points where the user showed an emotional response such as frustration. The group must then identify the functional issues or functional errors that were reported by most/all of the participants.
Given the small sample size of most usability tests, consistent usability problems found in most cases should be prioritized, and testing rerun with the proposed solution.
If all users can successfully complete the tasks, that indicates the product is usable, but does not indicate whether the product is desirable or whether the value proposition is actually delivered.

Potential Biases

Hawthorne effect (the observer effect): Users may behave differently when attempting to complete a task due to their awareness of being observed.
Social desirability bias: Users may try to answer questions or do tasks in a way that causes them to be viewed favorably by the experimenter.
Confirmation bias: Experimenters sometimes ask questions or create use cases in such a way that the user’s response/action confirms their preconceptions, hypothesis, or beliefs.
Selection bias: Selection of the incorrect audience can severely bias results. For example, testing usability with existing users will not show issues that new users may have with a product.

Field Tips

A usability test is the place to synthesize what you believe and what reality will accept - @ericries
The goal of a usability test is to make the users’ experience with the product easy and intuitive - @dharanidhar21
When testing usability, find users who are a little less savvy and aim to simplify your product - @TriKro
Got a tip? Add a tweetable quote by emailing us: [email protected]

Usability Testing

Usability Testing