Wizard of Oz

In Brief

A Wizard of Oz test is a product experience that looks fully built to the customer, but the response on the other end is generated by an operator behind the curtain — usually an LLM under human approval today, sometimes a person typing manually. The customer does not know. You collect real engagement data and qualitative feedback on a finished-feeling experience without committing to the underlying system. The output is two things at once: signal on whether the value proposition lands, and a transcript of what the “wizard” actually had to do — which becomes your spec for the real build.

Description

The Wizard of Oz technique was introduced at CHI ‘83 and formalized in J.F. Kelley’s 1984 ACM TOIS paper, which used it to prototype natural-language interfaces by having a human silently produce the system’s responses on the other side of a terminal. It stayed a niche HCI research method for the next forty years. LLMs changed the math: now the easiest way to fake an intelligent product is to wire it to an actual model and have a human review what comes out.

The 2026 version of the method is mostly a hybrid. The customer sees a real-looking interface. Behind it, an LLM generates the candidate response, and a human operator approves it, edits it, or rewrites it from scratch before it goes out. The operator is still the wizard — they’re just no longer typing every word.

This matters because it changes what you’re testing. A pure-AI prototype tests the model. A pure-human prototype tests the value proposition. A hybrid Wizard of Oz tests the value proposition and shows you exactly where the model would have failed if you had shipped the AI-only version — every override the operator makes is a labeled defect in the eventual automated build.

When to reach for Wizard of Oz vs. just shipping a real LLM-backed prototype:

• Use Wizard of Oz when getting a wrong answer to a real customer would be expensive or harmful, when the failure modes of the underlying system are unknown, when you want to count override rates as a quality bar, or when the system you eventually want is materially harder than what an off-the-shelf LLM can do.
• Use a Vibe-Coded Disposable MVP when an off-the-shelf LLM with a tight prompt is good enough that you don’t need a human in the loop, and you’d rather watch real users in a real product than read transcripts.
• Use a Concierge Test when you don’t yet know the right shape of the product — the customer should know a human is delivering the service, so you can ask “is this even useful?” before you fake the automation.

The legacy case — pure-human wizard, no LLM involved — is still valid in two situations: the system you’re testing is fundamentally not a language task (logistics, physical fulfillment, hardware control) and the customer-visible “magic” is a non-AI service; or you specifically want to evaluate a target capability that current models cannot reach, and you need a human ceiling to define what success would look like.

How to

Prep AI Prompt

1. State the value proposition you are testing.

Write it as a single sentence. “We help [customer] do [task] so they get [outcome] without [the part that’s expensive to build].” If you cannot finish that sentence, you are not ready to run a Wizard of Oz — you are still in problem discovery, and a Concierge Test is the better method.

2. Decide what’s faked and what’s real.

The wizard’s job is the one capability that would be expensive or risky to build. Everything else can be real or stitched together with no-code tools. List the parts of the experience and mark each: real, faked by wizard, or stubbed (placeholder). The fewer pieces the wizard is responsible for, the cleaner your signal.

3. Pick the wizard mode.

Choose one of three:

• LLM-assisted (default). An LLM drafts every response. The human operator reviews, edits, or rewrites before sending. Best for language-heavy products (advisors, summarizers, schedulers, agents).
• LLM-free. The operator types every response. Use this when the task is not a language task, when you want a clean human-ceiling baseline, or when LLM responses would be obviously wrong in this domain.
• Tool-augmented LLM. The LLM plus tool calls (search, calendar, database) drafts a response; the operator reviews. Use this when the eventual product needs to take actions, not just generate text.

4. Build the front-end.

Create whatever the customer will interact with: a web form, a chat interface, an app screen, a Slack bot, an email address. It should look like a finished product. Vibe-code it, use Typeform / Retool / Airtable / V0, or wire up a Slack workspace. The customer should not be able to tell that there’s a person in the loop — that includes latency cues. If your real product would respond instantly and your wizard takes three minutes, build in an explicit “thinking…” or “results within 5 minutes” affordance so the delay reads as design, not as a tell.

5. Build the operator console.

The operator needs three things on one screen: the incoming request, the LLM-drafted response (if applicable), and a single Approve / Edit / Reject control. Log every approval, every edit (with the diff), and every rejection. The edit log is the most valuable artifact this test produces — it’s a labeled dataset of where the eventual automation will fail.

6. Write the wizard playbook.

A two-page document the operator can read in five minutes. It defines:

• The persona the wizard is responding as (tone, voice, length, what they will and will not do).
• A response template for each request type you expect to see.
• An escalation rule: when does the wizard say “I don’t know” or “let me get back to you” rather than make something up?
• A safety rule: what categories of request must be rejected outright (medical advice, legal advice, anything outside the tested value proposition)?

If two operators will share shifts, the playbook is what keeps their behavior consistent.

Analysis AI Prompt

1. Read the override log first.

Every edit and rejection is a labeled defect. Cluster them: which request types did the LLM consistently get wrong? Which did the human have to write from scratch? Which would have been actively harmful if the customer had received the LLM’s first draft? This is the gap between “we have a working prototype” and “we have a real product.”

2. Compute the override rate.

Approve / Edit / Reject as a percentage of total responses. A 90% approve rate means an off-the-shelf LLM is probably enough — drop the wizard and run a Vibe-Coded Disposable MVP instead. A 30% approve rate means the human is doing most of the work — the eventual product is a much harder build than you thought, and you need to decide if the value proposition is strong enough to justify it.

3. Measure the real behavior.

Did customers come back? Did they pay? Did they refer? Compare to the assumption you wrote down in Prep step 1. Surface-level satisfaction does not validate the value proposition; behavioral commitment does.

4. Estimate the cost of automation.

For each cluster of requests in the override log, estimate what it would take to build that capability: a better prompt, a fine-tune, a tool call, a custom model, or “we cannot build this in the next 12 months.” Multiply by frequency. The clusters that are common AND expensive to automate are where your engineering risk lives.

5. Decide the next move.

One of three things should happen:

• The override rate is low, customers came back, the real behavior is there → ship a real build, no wizard.
• The override rate is high but the value proposition lands → keep the wizard, scale operators, treat the human-in-the-loop as a feature for now.
• The override rate is irrelevant because the real behavior didn’t show up → the magic isn’t real. Stop. Go back to a Concierge Test and figure out what customers actually want.