Quick Answer: Most founders get bad data not because they ran the experiment badly, but because they ran the wrong experiment for the question they actually had. The Real Startup Book sorts any question by two cuts — market or product (problem or solution), generative or evaluative (open-ended, or a hypothesis you can prove yes or no) — into one of four quadrants, each with its own methods. AI can sort a question into a quadrant and run the resulting experiment ten times faster. What it cannot do is decide which question is worth asking in the first place. That part is still yours.

This is the first of a short teaser series for the second edition of The Real Startup Book. Each post pulls one chapter forward and shows what changed (and what stubbornly did not) now that AI is in every founder’s toolkit.

The mistake AI made cheaper

The Real Startup Book has always opened the same way: before you pick a method, find the right question. The most common failure mode we saw — across accelerator cohorts, corporate innovation teams, and solo founders — was running the wrong experiment for the question on the table — the problem we built a whole decision framework around more than a decade ago.

A founder who is uncertain whether anyone has the problem they think they are solving runs a landing-page smoke test, gets a 2% click-through, and concludes nothing. The smoke test was the wrong experiment. The right one was a customer interview. A founder who is uncertain about pricing builds the whole product first and is shocked when nobody pays. The right experiment was a pricing probe before the build.

This is not hypothetical. Lean Foundry documented a founder who ran fifty interviews on a VR rendering tool and came away with a spreadsheet of two hundred-plus problems and still could not say what to build. A single thirty-minute conversation about one recent client presentation surfaced the real problem — $400 and a three-day wait for every revision. That moved the product’s value proposition from “better rendering” to instant self-serve revisions. Fifty interviews with no sharp hypothesis produced noise; one question aimed at a specific past event produced the pivot.

AI did not invent this mistake. AI made it cheaper, faster, and more confident-sounding.

In 2015 the cost of running the wrong experiment was a couple of weeks. In 2026 a founder with Claude or Cursor can spin up the wrong experiment in an afternoon, get a beautifully-formatted report on it, and feel like they made progress. The methods are the same. The penalty for skipping the diagnostic is now hidden under a thicker layer of polish.

The diagnostic: which experiment answers your question

The diagnostic in The Real Startup Book is a 2×2. Before you pick a method, you place your question on two axes:

1. Market or product? Are you still uncertain about the customer and their problem (market), or about the solution that would fix it (product)?
2. Generative or evaluative? Do you have a specific, falsifiable hypothesis you can prove yes or no (evaluative)? Or is the question still open-ended, and you need to generate ideas first (generative)?

Those two cuts land you in one of four quadrants, each with its own methods:

• Generative market research — who is the customer, what is their pain? Customer interviews, data mining.
• Evaluative market experiments — will this segment click, will they pay? The smoke test.
• Generative product research — what solution, what minimum feature set? Concierge tests, solution interviews.
• Evaluative product experiments — does the solution actually work? Wizard of Oz, usability testing.

The single most common mistake is running an evaluative experiment when you are actually in generative territory: firing off a smoke test when you do not yet have a sharp hypothesis or a defined customer. The data comes back ambiguous — was the 2% because no one wants it, or because the wrong people saw a confusing headline? — and you learn nothing.

And one rule the book will not bend on: priority is singular. Pick the one question whose wrong answer would do the most damage to the business, and answer that one first.

What AI is and is not useful for here

Draw the line in the right place and AI earns its keep on both sides of it.

Once you have a question, AI can help you place it on the 2×2 — market or product, generative or evaluative — and name the method that fits. Once you know what you are testing, AI can draft the landing-page copy, generate the interview guide, scaffold the analytics, summarize the transcripts, and write the read-out. The execution work compresses by an order of magnitude. AI can even help at the fuzzy front end — talking through your idea to surface assumptions you had not written down, and pressure-testing them from a few angles.

What AI cannot do is decide which question is worth asking. That call depends on things that are not in the model: what your customers actually said, what you saw when you watched them work, the tacit sense of where your business is most likely to be wrong. The book’s AI prompts are built to draw the assumptions out of you, not to decide for you. Ask a model to just pick, and it will cheerfully hand you a customer segment, a problem, a metric — all plausible, but completely ungrounded in reality. It will then run whatever experiment you asked for, whether or not it was the one you needed.

So the division of labor is the opposite of what it looks like. You can hand off a lot of the grunt work of running an experiment to AI. But the part that feels like it should be easy — deciding what to ask — is the one only you can do.

Read the chapter

Asking the Right Question covers how to surface the questions in the first place, pressure-test them against your blind spots, and prioritize down to the one that matters. From Question to Method walks through the 2×2 that turns that question into the right experiment. And once you are running an evaluative experiment, the hypothesis checklist makes the hypothesis testable — one change, one expected impact, one metric, one timeframe.

Next post in the series: the smoke test. Same experiment we have been running since 2015, now an order of magnitude cheaper to run badly.

— Tristan

Comments

Loading comments…

Leave a comment

Name *

Email * (not published)

Website (optional)

Comment *

Leave this blank

Post Comment