For more ways to test your solution or figure out your minimum viable feature set, download the Real Startup Book which is co-created by 40+ lean startup enthusiasts.

Wizard of Oz Prototyping

wizard of oz prototyping - Pay no attention to the man behind the curtain

In the lean startup world, the wizard of oz prototyping method was used by companies like Aardvark and CardMunch. Both were able to create amazing effective prototypes very quickly by not building anything.

Aardvark connected people with questions to people with expertise. Cardmunch managed to transcribe blurry photos of business cards better than any other Optical Character Recognition (OCR) system at the time. Neither had any technology or algorithm behind them.

The technology was human beings.

Cardmunch leveraged Amazon’s Mechanical Turk while Aardvark used interns. Instead of building complex algorithms both just faked it. Since neither needed a real-time response to provide their core value propositions, there was no need to build anything.

To the user, the products delivered exactly what they promised. A little slow perhaps…but the value was there.

(Note: Lean Startup did not invent this technique. It was already being used in the lab by 1975 by J. F. Kelley.)

Concierge Test

Concierge test bellConcierge testing was used by companies such as Wealthfront and Food on the Table. The approach is similar in that there is no technology involved. Both companies performed the service they were offering as a concierge at a hotel might. Purely manually.

Wealthfront would work with its customers to provide wealth management advice and portfolio management. They would sit with them, pen and paper in hand, and figure out solutions to wealth management issues step by step.

Manuel Russo used to personally go shopping with his customers to help them plan their meals. (Not very scalable!)

So both Wizard of oz prototyping and Concierge testing provide the value proposition manually without any technology. What’s the difference?

User Experience

The difference is that in one case, the user knows about the concierge, while the Wizard of Oz is hidden from the user. This is a key difference for the user experience.

In a Wizard of Oz test, the user experience is a complete simulation of the actual intended product, but with a slight time delay since a human cannot react as fast as a computer. The delivered value proposition is almost identical but slightly under performing.

In a Concierge test, the value proposition is actually significantly higher than the eventual product. That’s because the customer is aware and is interacting with a real human being.

Humans are Social Creatures

humans are social creaturesHuman interaction, as a whole, is generally a positive value proposition. We’re social creatures and even introverts like me gather in groups to socialize.

There are some circumstances where we do not want to interact with a human being. For me, I don’t want to talk to anyone at Macy’s. Malls and department stores just kind of freak me out in general.

For everyone else, just imagine that you just typed a search into WebMD to do a little private research and all of a sudden a window pops up with a live human being on a webcam to tell you all about your search query for genital herpes. Fun!

Compare that to Food on the Table where we’d get to go shopping with Manuel Russo. Normally, I find shopping a bit of a chore, but Manuel is a pretty fun guy. I was delighted to meet him a few years ago in Austin and I would be equally delighted to go shopping with him. (Manuel seems to know all the best places to get food in the city and is hysterically funny.)

A Concierge test has an extra value proposition that a Wizard of Oz test does not. Or at the very least, the test is severely affected by the visible presence of a human being.

Generative vs. Evaluative

This brings us back to the distinction of Generative Research vs. Evaluative Experiments.

Generative Research vs. Evaluative Experiments

Wizard of Oz testing allow us to test a clearly defined hypothesis about what solution will provide the value proposition to the customer. We test it by simulating the user experience as precisely as possible without technology. The lack of technology means we can rapidly test a variety of hypothesis and rapidly iterate.

Since Wizard of Oz allows us to test and falsify a solution hypothesis, it’s an Evaluative Product Experiment.

Concierge testing has a huge human factor which biases the test. If we attempt to use a Concierge test to validate our solution hypothesis, we’re liable to get a bias towards a false positive or perhaps even to a false negative in some circumstances. It’s simply the wrong type of test to falsify a clear hypothesis.

Concierge testing is very bad when we have a clear idea of the solution, but very good when we’re not exactly sure what the solution should be!

Generating Ideas

generate ideasWhen Wealthfront sat down with their customers, they had some idea on how to manage finances, but they didn’t have a step 1, 2, 3 algorithm of how to do it.

There were many unknowns including how to best represent the portfolio information to the customer in an intelligible way to them to make their own decisions.

Concierge testing allows us to work with the customer without having a clear solution hypothesis. A vague idea, or even no idea, will sometimes be enough to muddle through the process while talking with the customer and getting constant feedback.

We might think that it’s impossible to go to a customer and just sell the promise of a solution without having any clear idea of how to provide it. It’s not.

Selling solutions that don't exist and then winging it is common place. It's called Consulting. Share on X

(Note: Lean startup didn’t come up with Concierge testing either. Consulting has been around since the before the pharaohs.)

consultant to the pharoahs - a.k.a concierge test

By the time we’ve manually performed the task a few times for our consulting clients, we have a much clearer idea of what form an automated solution can take and what the minimum viable feature set looks like.

The solution is not yet validated, but we can take that hypothesis and begin the process of evaluating experimentation with a Wizard of Oz test, paper prototyping, usability tests, etc.

wizard of oz prototyping vs concierge test

Lessons Learned

Wizard of Oz hides the human. Concierge flaunts the human. Share on X The Wizard of Oz method is for evaluating a solution hypothesis. Share on X The Concierge method is for generating solutions ideas and hypotheses. Share on X

Confusing them can lead to faulty reasoning and flawed business models.

For more ways to test your solution or figure out your minimum viable feature set, download the Real Startup Book which is co-created by 40+ lean startup enthusiasts.

Make better product and business decisions with actionable data


Gain confidence that you're running the right kind of tests in a five-week series of live sessions and online exercises with our Running Better Experiments program. Refine your experiment process to reduce bias, uncover actionable results, and define clear next steps.
Join the waitlist →