6.5Usability Testing

In Brief

Description

Usability testing is a qualitative method for observing real people as they attempt realistic tasks with your product. ISO 9241-11:2018 defines usability as the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. The test makes that definition operational: you put a representative user in front of the artifact, give them a goal, and watch where reality diverges from the design intent. Jakob Nielsen’s foundational Usability Engineering established that you do not need a large sample to find most usability problems — five carefully chosen testers, run iteratively, will surface the dominant defects in a flow.

The method works at any fidelity. A paper sketch tested with three people can expose a confusing label as effectively as a fully built product tested with twenty. Steve Krug’s Don’t Make Me Think, Revisited makes the case for treating usability testing as an everyday DIY activity rather than a heavy lab exercise: small, frequent tests done by the team that built the product close the loop faster than commissioning quarterly studies. Usability testing is qualitative — it tells you what breaks and why — and complements quantitative methods (analytics, A/B tests, surveys) that tell you how often.

AI now sits inside the workflow rather than alongside it. Platforms like Maze, Lookback, Lyssna, and UserTesting auto-generate session transcripts, flag moments of hesitation or frustration from screen and audio signals, cluster patterns across sessions, and produce draft summaries. This compresses analysis time from hours to minutes and makes it practical to run more tests with more users. AI does not replace observation — it accelerates synthesis. A flagged frustration moment still needs a human to decide whether the design is wrong or the task scenario was unfair, and the empathy gain that comes from watching a user struggle live is something a transcript summary cannot reproduce.

How to

Prep AI Prompt

1. Define the task scenarios. Write 3–5 realistic tasks that cover the critical paths in your product. Each task is a goal, not a click-by-click instruction. Example: “You just signed up. Find the feature that lets you invite a teammate.” Avoid leading the user toward the answer. Set a reasonable time limit per task (2–5 minutes).
2. Recruit 5 testers who match your target user. Five is the heuristic Nielsen established in Usability Engineering — past five, you mostly re-confirm problems you’ve already found. Recruit from existing users, social channels, or panels (UserTesting, Lyssna, Prolific). For B2B, ask customers directly; most will say yes if you make it short.
3. Decide moderated vs unmoderated, then pick the tool. Moderated sessions (Zoom, Lookback, in-person) let you ask “what made you click that?” in real time but cost more researcher hours. Unmoderated sessions (Maze, Lyssna, UserTesting) scale cheaply and AI-assisted platforms can analyze think-aloud audio and behavioral signals across many sessions without a researcher watching each one in real time. Mix both when budget allows: a few moderated sessions to build empathy, a larger unmoderated batch to confirm patterns.
4. Write a short intro script. Two or three sentences that frame the session. The standard reassurance: “We’re testing the product, not you. If you get stuck, that’s exactly the feedback we need.” Pre-writing the script keeps facilitators consistent across sessions.
5. Pilot once internally before the first real session. Run the full protocol with a teammate. You’re checking whether tasks are intelligible, whether the recording setup works, and whether the timing fits. Half the issues you find in piloting would have wasted a real participant slot.
6. Decide who moderates and who takes notes. One person facilitates and asks the questions; a second person observes and captures notes. Trying to do both at once degrades both. For unmoderated tests, the “observer” role becomes whoever reviews the AI-generated session summaries and watches the flagged moments.

Analysis AI Prompt

1. Synthesize observation notes across sessions. The facilitator and any observers compare notes, focusing on moments where users hesitated, backtracked, or expressed frustration. Even usability experts sometimes disagree on interpretation, so multiple observers reduce the chance that a single experimenter’s bias filters the findings. Identify the functional issues that affected most or all participants — those are the ones worth fixing first.
2. Prioritize by frequency and severity. Given the small sample size, treat consistent problems (3+ of 5 users hit the same wall) as confirmed defects. One-off issues might be real or might be participant-specific — note them but do not over-weight them. Severity is independent of frequency: a problem that happens once but blocks the user from completing the task scores higher than a cosmetic issue that everyone hit.
3. Use AI tools to triage what to watch closely. AI-powered session analysis tools (Maze, Lookback, Lyssna) auto-flag moments of confusion, generate heatmaps, calculate task-completion metrics, and produce session summaries. Treat these as a multiplier on researcher bandwidth — use them to triage which recordings to watch closely, not as a substitute for watching sessions yourself. Automated heuristic checks (contrast ratios, touch-target sizes, navigation depth) can also catch surface-level issues before you spend a real participant slot on them.
4. Distinguish usable from desirable. If all users complete the tasks, that tells you the product is usable. It does not tell you whether the value proposition lands or whether anyone would actually pay for it. Usability testing is a necessary but insufficient validation step — pair it with desirability and value-proposition tests before you assume you are clear to ship.