6.7 Wizard of Oz

In Brief

Description

A Wizard of Oz test is a product experiment in which a human manually performs the work behind an interface the user believes is automated. The user interacts with what looks like a finished product; an operator hidden behind the interface produces the responses. You measure how the user behaves on a finished-feeling experience without building the system that would deliver it.

The common form today is a hybrid. The user sees a real-looking interface. Behind it, an LLM generates the candidate response, and a human operator approves it, edits it, or rewrites it from scratch before it goes out. The operator is the wizard, even when they are no longer typing every word.

This matters because it changes what you’re testing. A pure-AI prototype tests the model. A pure-human prototype tests the value proposition. A hybrid Wizard of Oz tests the value proposition and shows you exactly where the model would have failed if you had shipped the AI-only version — every override the operator makes is a labeled defect in the eventual automated build.

When to reach for Wizard of Oz vs. just shipping a real LLM-backed prototype:

• Use Wizard of Oz when getting a wrong answer to a real user would be expensive or harmful, when the failure modes of the underlying system are unknown, when you want to count override rates as a quality bar, or when the system you eventually want is materially harder than what an off-the-shelf LLM can do.
• Use a Vibe-Coded Disposable MVP when an off-the-shelf LLM with a tight prompt is good enough that you don’t need a human in the loop, and you’d rather watch real users in a real product than read transcripts.
• Use a Concierge Test when you don’t yet know the right shape of the product. In a Concierge Test the user knows a human is delivering the service, so you can ask “is this even useful?” before you hide the work behind a fake automation.

The pure-human wizard — no LLM involved — still fits two situations: the system you’re testing is fundamentally not a language task (logistics, physical fulfillment, hardware control) and the user-visible “magic” is a non-AI service; or you want to evaluate a target capability that current models cannot reach, and you need a human ceiling to define what success would look like.

Ethics — The user believes they are using an automated product when a human is doing the work, and if you charge them, they are paying real money on that belief. Keep the deception proportionate: do not let the wizard give advice a real person would act on in high-stakes domains (medical, legal, financial), refund anyone who paid for a capability you have not built, and never collect or retain data you would not be allowed to collect in the shipped product.

How to

Prep AI Prompt

1. State the value proposition you are testing.

Write it as a single sentence. “We help [user] do [task] so they get [outcome] without [the part that’s expensive to build].” If you cannot finish that sentence, you are not ready to run a Wizard of Oz — you are still in problem discovery, and a Concierge Test is the better method.

2. Decide what’s faked and what’s real.

The wizard’s job is the one capability that would be expensive or risky to build. Everything else can be real or stitched together with no-code tools. List the parts of the experience and mark each: real, faked by wizard, or stubbed (placeholder). The fewer pieces the wizard is responsible for, the cleaner your signal.

3. Pick the wizard mode.

Choose one of three:

• LLM-assisted (default). An LLM drafts every response. The human operator reviews, edits, or rewrites before sending. Best for language-heavy products (advisors, summarizers, schedulers, agents).
• LLM-free. The operator types every response. Use this when the task is not a language task, when you want a clean human-ceiling baseline, or when LLM responses would be obviously wrong in this domain.
• Tool-augmented LLM. The LLM plus tool calls (search, calendar, database) drafts a response; the operator reviews. Use this when the eventual product needs to take actions, not just generate text.

4. Build the front-end.

Create whatever the user will interact with: a web form, a chat interface, an app screen, a chat-workspace bot, an email address. It should look like a finished product. Vibe-code it, use no-code form and app builders, or wire up a bot in your team chat tool. The user should not be able to tell that there’s a person in the loop — that includes latency cues. If your real product would respond instantly and your wizard takes three minutes, build in an explicit “thinking…” or “results within 5 minutes” affordance so the delay reads as design, not as a tell.

5. Build the operator console.

The operator needs three things on one screen: the incoming request, the LLM-drafted response (if applicable), and a single Approve / Edit / Reject control. Log every approval, every edit (with the diff), and every rejection. The edit log is the most valuable artifact this test produces — it’s a labeled dataset of where the eventual automation will fail.

6. Write the wizard playbook.

A two-page document the operator can read in five minutes. It defines:

• The persona the wizard is responding as (tone, voice, length, what they will and will not do).
• A response template for each request type you expect to see.
• An escalation rule: when does the wizard say “I don’t know” or “let me get back to you” rather than make something up?
• A safety rule: what categories of request must be rejected outright (medical advice, legal advice, anything outside the tested value proposition)?

If two operators will share shifts, the playbook is what keeps their behavior consistent.

Analysis AI Prompt

1. Read the override log first.

Every edit and rejection is a labeled defect. Cluster them: which request types did the LLM consistently get wrong? Which did the human have to write from scratch? Which would have been actively harmful if the user had received the LLM’s first draft? This is the gap between “we have a working prototype” and “we have a real product.”

2. Compute the override rate.

Approve / Edit / Reject as a percentage of total responses. A 90% approve rate means an off-the-shelf LLM is probably enough — drop the wizard and run a Vibe-Coded Disposable MVP instead. A 30% approve rate means the human is doing most of the work — the eventual product is a much harder build than you thought, and you need to decide if the value proposition is strong enough to justify it.

3. Measure the real behavior.

Did users come back? Did they pay? Did they refer? Compare to the assumption you wrote down in Prep step 1. Surface-level satisfaction does not validate the value proposition; behavioral commitment does.

4. Estimate the cost of automation.

For each cluster of requests in the override log, estimate what it would take to build that capability: a better prompt, a fine-tune, a tool call, a custom model, or “we cannot build this in the next 12 months.” Multiply by frequency. The clusters that are common AND expensive to automate are where your engineering risk lives.

5. Decide the next move.

One of three things should happen:

• The override rate is low, users came back, the real behavior is there → ship a real build, no wizard.
• The override rate is high but the value proposition lands → keep the wizard, scale operators, treat the human-in-the-loop as a feature for now.
• The override rate is irrelevant because the real behavior didn’t show up → the magic isn’t real. Stop. Go back to a Concierge Test and figure out what users actually want.