Evaluative Research: Why Talking to Customers Isn't Enough
Six people saw my mock-up. Nobody noticed the missing contact button.
By Tristan Kromer · Quick Answer: Evaluative research means letting your product speak for itself instead of walking customers through it. As product managers, we often mistake “talking to customers” for genuine feedback — but guided walkthroughs mask critical problems. By asking non-leading questions like “What do you think this page does?” and simply listening, you’ll uncover issues that even multiple review rounds miss. In one case, six people reviewed a mock-up without noticing it lacked a basic contact button — revealed only after switching to a listen-first interview technique.
I realized today that I have been taking the lean startup / customer development axiom of “talk to customers” far too literally. While the quality of my feedback (evaluative research) has been reasonably good and I’ve gotten a number of good ideas, there’s a big difference between talking to customers and listening to customers. I’ve been having far to much of a two way dialog and not letting my product speak for itself.
Screen-shot Demo
I got a call today from Performable.com which is developing a new A/B testing tool for this type of evaluative research. I signed up for their beta test a few weeks ago and they gave me a call to ask if I would be willing to walk through a demo with them. Within a minute, they had a screen share up and were showing me a live demo. Although I had no control, it was definitely not a walk through in the traditional sense. it wasn’t a guided tour so much as an exploration. They showed me a screen and started peppering me with questions. Some of my favorites:
- What do you think this page does?
- What do you think will happen if you click this button?
- What do you think this wording means?
Within a few minutes they had me dictating every thought running to my head and quantifying exactly what I needed from an A/B test tool. At no point did they resort to telling me what the page was supposed to do. In truth, I don’t really even know if the product worked or if I was just looking at static pages. That’s exactly the sort of feedback I need.
Big Ears
Getting back to work I decided to immediately get some customers and ask them about my Photoshop mock-up of a user’s profile page. This is the mock-up that I thought was “good enough” and that both my partners, a professional designer, and two customers had already looked over. At least for version 0.1, I thought my customer development was done. I was wrong. One of the very first things that the customer asked was, “How do I contact this person?” After spending more time on this than I’d care to admit and four sets of eyes staring at this page multiple times, we had forgotten the most basic functionality of all! There was no space for a contact / connect button. Doh! Of course, customer development is an ongoing process and there is no “done"", but this was an embarrassing omission and spoke to something more fundamentally wrong in my customer interview technique. The big mistake was that I had previously performed more of a walk through, explaining each section to users and guiding them through. This time I said less, and listened more. I didn’t offer any leading questions until they stopped talking, and then I tried to say very little aside from, “What about this?” “What about this section?” I won’t go into the rest of the embarrassing elements I missed, but the change in customer interview style was very rewarding. So kudos and thanks to the Performable.com team!
Frequently Asked Questions
What is evaluative research in customer development?
Evaluative research is the process of gathering feedback by observing how customers interact with and interpret your product or mock-up. Rather than explaining what your product does, you let the product speak for itself by asking open-ended questions like “What do you think this page does?” — then listening carefully to uncover gaps and misunderstandings.
How do you avoid leading customers during product feedback sessions?
As product managers, we need to resist the urge to guide customers through our designs. Instead of explaining each section, ask non-leading questions like “What do you think will happen if you click this?” and let the customer talk. Only prompt with minimal nudges like “What about this section?” once they stop sharing their thoughts. The goal is listening, not having a two-way dialog.
Why is showing a mock-up not enough for customer development?
Simply showing a mock-up to people and getting a thumbs-up isn’t real evaluative research. In the article, six people — including partners, a designer, and customers — reviewed a mock-up and missed that it lacked a basic contact button. The difference was in how the feedback was gathered: guided walkthroughs mask problems, while open-ended observation reveals critical omissions.
What are the best questions to ask during evaluative research?
Some of the most effective questions include: “What do you think this page does?”, “What do you think will happen if you click this button?”, and “What do you think this wording means?” These questions force the customer to interpret the product on their own, revealing whether your design actually communicates what you intend — without you biasing their response.
When is customer development actually “done”?
It’s not. Customer development is an ongoing process, and assuming it’s “done” — even for a version 0.1 — is a common mistake. As the article demonstrates, switching from a guided walkthrough to a listen-first interview technique immediately uncovered fundamental issues that multiple rounds of prior review had missed. There’s always more to learn from customers.
Comments
Loading comments…
Leave a comment