Evaluative Research Of Customer Development Mistakes

I realized today that I have been taking the lean startup / customer development axiom of “talk to customers” far too literally.While the quality of my feedback (evaluative research) has been reasonably good and I’ve gotten a number of good ideas, there’s a big difference between talking to customers and listening to customers.

I’ve been having far to much of a two way dialog and not letting my product speak for itself.

Screen-shot Demo

I got a call today from Performable.com which is developing a new A/B testing tool for this type of evaluative research. I signed up for their beta test a few weeks ago and they gave me a call to ask if I would be willing to walk through a demo with them.

Within a minute, they had a screen share up and were showing me a live demo. Although I had no control, it was definitely not a walk through in the traditional sense. it wasn’t a guided tour so much as an exploration. They showed me a screen and started peppering me with questions. Some of my favorites:

  • What do you think this page does?
  • What do you think will happen if you click this button?
  • What do you think this wording means?

Within a few minutes they had me dictating every thought running to my head and quantifying exactly what I needed from an A/B test tool. At no point did they resort to telling me what the page was supposed to do. In truth, I don’t really even know if the product worked or if I was just looking at static pages.

That’s exactly the sort of feedback I need.

Big Ears

Getting back to work I decided to immediately get some customers and ask them about my Photoshop mock-up of a user’s profile page. This is the mock-up that I thought was “good enough” and that both my partners, a professional designer, and two customers had already looked over. At least for version 0.1, I thought my customer development was done.

I was wrong.

One of the very first things that the customer asked was, “How do I contact this person?”

After spending more time on this than I’d care to admit and four sets of eyes staring at this page multiple times, we had forgotten the most basic functionality of all! There was no space for a contact / connect button. Doh!

Of course, customer development is an ongoing process and there is no “done””, but this was an embarrassing omission and spoke to something more fundamentally wrong in my customer interview technique. The big mistake was that I had previously performed more of a walk through, explaining each section to users and guiding them through.

This time I said less, and listened more. I didn’t offer any leading questions until they stopped talking, and then I tried to say very little aside from, “What about this?” “What about this section?”

I won’t go into the rest of the embarrassing elements I missed, but the change in customer interview style was very rewarding. So kudos and thanks to the Performable.com team!