Kromatic logo

The Real Startup Book

Evidence-based guides and research methods from Kromatic

5.4 Competitor Usability Testing

An observer with a clipboard watching a user complete tasks on a laptop

At a Glance

QualitativeIn-personRemoteObservational
~3 days–2 weeks
$0–$300

Other names Competitor UX Test · Comparative Usability Test

In Brief

Watch target participants complete tasks using a competitor’s product, and note where they struggle, what workarounds they invent, and which features they ignore. The output is a set of pain points, unmet needs, and unnecessary features that directly inform the design of your own product. See Usability Testing for testing your own product once you have one.

Common Use Case

You are designing a new product and want to learn where people struggle with the existing alternatives before you commit to your own design. You recruit a few participants who match your target customer, watch them complete real tasks in a competitor’s product, and note every point of confusion, workaround, or complaint. The gaps you uncover tell you where your solution can differentiate.

Helps Answer

  • What is the minimum feature set needed to solve the problem?
  • How important is design quality to users in this category?
  • Where do people struggle most with existing products?
  • Which competitor features are unnecessary or confusing?

Description

Competitor usability testing is a structured observation method where you watch target participants attempt real tasks in a competitor’s product, or a substitute good, and note where they get stuck. You are mining that product for ideas: watching where the existing alternative fails the people who use it tells you what your own product should do better. The procedure is the same as testing your own product, but the purpose is generative rather than evaluative. You are not verifying that a product works and you have no product of your own to benchmark — the competitor’s product is the raw material for your design.

The competitor does not have to be a direct rival. To generate ideas for a better U.S. tax experience, you could watch participants prepare taxes in Sweden or India. The results would not tell you whether the U.S. tax experience is good, but they may suggest whether to focus on the comprehensibility of the tax code, the submission process, or the rules themselves.

How to

Prep

  1. Pick 2–3 competitor products or substitutes to test. Choose the ones your target participants are most likely using today. Include at least one indirect substitute — a different type of product that solves the same problem in a different way. Keep the count to four or fewer; beyond that, the analysis cost grows faster than the insight does.

  2. Write 4–6 task scenarios. Each task should represent a core workflow your participant does regularly. Use the same tasks across all competitors so you can compare. Frame each as a goal, not a click path: “You need to [accomplish goal]. Use [Competitor A] to do it.” Keep each scenario short, specific, and free of jargon the participant would not use. To find candidate usability issues worth turning into scenarios, summarize a competitor’s app-store reviews, support forums, and social-media complaints — treat the output as a list of things to watch for, not a finished answer.

  3. Recruit 5 participants per competitor. They should match your target customer profile. If a participant already uses one of the products, that is fine — you will learn where experienced users still struggle. Five per product is the point at which most usability problems on a single product surface.

  4. Decide how you will run the sessions. Choose between moderated sessions (you watch live and ask questions) and unmoderated ones (the participant records themselves alone), and between in-person and remote, then pick the tool to match (see Tools below). For unmoderated remote tests, pre-record a short framing message; for moderated tests, write a one-paragraph intro. Keep the framing identical across competitors so participants approach each product the same way.

  5. Write your consent and recording language, and your task prompts. Get explicit permission to record. Make clear you are testing the products, not the participant, and that they can quit a task or the whole session at any time without explanation. Decide the questions you will ask after each task and at the end of the session now, so every session runs the same way.

Execution

  1. Run the test. For each participant and each product:

    • Give them the task. Don’t explain how the product works.
    • Watch them attempt the task. Note where they hesitate, make errors, express frustration, or find a workaround.
    • Ask your prepared post-task and end-of-session questions, the same way every time.

  2. Record and compare. For each product, track how often participants finished the task, how long it took, how many errors they made, and how satisfied they were. The gaps and frustrations you observe are your opportunity — they reveal what your product should do better.

For the full usability testing methodology, see Usability Testing. The process is the same; the difference is that you are testing someone else’s product to generate ideas, not evaluating your own.

Analysis

  1. Treat results as generative, not evaluative. A failed task on a competitor’s product is a candidate problem to solve, not a verdict on the competitor. The ideas tend to be unstructured and piecemeal — you will need to integrate them into a coherent solution rather than ship them one-for-one.

  2. Cluster failures into opportunity areas. Group every observed pain point, workaround, and unmet need across participants and products. The clusters with the most repeats across participants are your highest-confidence opportunities. A pain point that appeared once is an anecdote; one that appeared in four of five sessions across two competitors is a pattern.

  3. Validate before building. Before committing engineering effort to any opportunity, test it with a generative product method such as a Solution Interview or Concierge Test. Keep validating the problem for as long as it is cheap to do so, and resist building a feature just because a competitor lacks it — confirm the problem is worth solving first.

Biases & Tips
  • Observer effect Participants behave differently when they know they are being watched. Frame the session as testing the product, not the person, and resist coaching mid-task.
  • Confirmation bias You may write tasks or follow-up questions in a way that confirms what you already believe about a competitor’s weakness. Have a second person review your task list and questions before you run sessions.
  • Selection bias Recruiting whoever is easiest to reach, or only people who already dislike the competitor, distorts what you see. Recruit against your target customer profile and include both current users of the product and newcomers to it.
  • Curse of knowledge You know the problem domain deeply, so failures that feel obvious to you may be the participant’s real struggle. Resist explaining how the product is “supposed” to work; the confusion is the data.
  • AI analysis substitution A model-generated summary of session recordings is not a substitute for watching the sessions. Watch at least the first three sessions yourself before relying on any tooling to surface the rest.

Next Steps

  • Map the most common usability failures to specific feature opportunities in your product.
  • Use competitor pain points in your marketing messaging (“Unlike X, we…”).
  • Test your own product or prototype with the same tasks to benchmark against competitors.
  • Prioritize building features that address the highest-frustration competitor workflows.
  • Use a Solution Interview to propose your alternative solution and gauge whether it addresses the competitor pain points you discovered.
  • Run Usability Testing on your own prototype using the same tasks to verify your design actually outperforms the competition.
Learn more

Case Studies

American Airlines: Comparative QXscore testing

American Airlines worked with UserTesting to score four customer journeys (booking, flight changes, check-in, AAdvantage) against the same flows on competing airlines, surfacing friction like below-the-fold multi-city search; redesigns produced a 37% lift in task completion and a 15% average QXscore increase.

Read more

Loop11: Ten-airline banned-baggage benchmark

Loop11 recruited 1,000 participants (100 per site) to attempt the same banned-baggage lookup task across ten airline websites; British Airways completed at 71% in 87 seconds, while Malaysia Airlines completed at 31% and Virgin Atlantic took 199 seconds — more than twice as long as BA.

Read more

Further reading

  • Jakob Nielsen, Usability Engineering. Academic Press, 1993. ISBN 978-0125184052. The foundational text for the small-sample testing convention used here.
  • “Competitive Usability Evaluations: Definition”
  • “Tools for Unmoderated Usability Testing”
  • Steve Krug, Don’t Make Me Think, Revisited (3rd ed.). New Riders, 2014. ISBN 978-0321965516. Practical guidance on writing usability test tasks that produce real signal.
  • Tomer Sharon, Validating Product Ideas: Through Lean User Research. Rosenfeld Media, 2016. ISBN 978-1933820293. Lean research framing for treating session output as hypotheses to validate.
  • Dan Olsen, The Lean Product Playbook. Wiley, 2015. ISBN 978-1118960875. Problem-space-before-solution-space discipline for opportunities surfaced from competitor sessions.
  • Daniël De Wit: Don’t feel bad, user test your competitors

Got something to add? Share with the community.