Before You Run the Experiment, Run the Hypothesis Checklist
RSB 2nd Edition teaser: the diagnostic step AI cannot do for you.
By Tristan Kromer · Quick Answer: Most founders pick the wrong experiment because they never wrote the hypothesis down. The hypothesis checklist in The Real Startup Book is a five-question diagnostic — who is the customer, what problem do they have, what solution are you testing, how will you know it worked, and how will you know it failed — that decides which method to run next. AI is great at executing an experiment ten times faster. It is not great at telling you which experiment to run. The checklist is the part you have to do yourself, and it takes about fifteen minutes.
This is the first of a short teaser series for the second edition of The Real Startup Book. Each post pulls one chapter forward and shows what changed (and what stubbornly did not) now that AI is in every founder’s toolkit.
The mistake AI made cheaper
The first edition of The Real Startup Book opened with the hypothesis checklist for a reason. The most common failure mode we saw — across accelerator cohorts, corporate innovation teams, and solo founders — was running the wrong experiment for the question on the table.
A founder who is uncertain whether anyone has the problem they think they are solving runs a landing-page smoke test, gets a 2% click-through, and concludes nothing. The smoke test was the wrong experiment. The right one was a customer interview. A founder who is uncertain about pricing builds the whole product first and is shocked when nobody pays. The right experiment was a pricing probe before the build.
AI did not invent this mistake. AI made it cheaper, faster, and more confident-sounding.
In 2015 the cost of running the wrong experiment was a couple of weeks. In 2026 a founder with Claude or Cursor can spin up the wrong experiment in an afternoon, get a beautifully-formatted report on it, and feel like they made progress. The methods are the same. The penalty for skipping the diagnostic is now hidden under a thicker layer of polish.
The five questions
The checklist is short on purpose. If you cannot answer these five in writing, you are not ready to run an experiment yet:
- Who is the customer? Specific enough that you could find ten of them this week. “Small business owners” is not a customer. “Plumbing-company owners in the Bay Area who quote jobs from their truck” is.
- What problem do they have? A problem the customer would describe in their own words, unprompted. Not a feature you wish they wanted.
- What solution are you testing? The minimum thing you need to put in front of them to learn whether the problem-solution pairing works.
- How will you know it worked? The specific signal — a number, a behavior, a quote — that would make you continue.
- How will you know it failed? The specific signal that would make you stop. Written down before you run it. This is the question founders skip most often and the one that costs the most when skipped.
If question 5 is “we’ll see how it goes,” you are not running an experiment. You are running a vanity exercise that will produce a story you can tell yourself either way.
What AI is and is not useful for here
AI is great at the parts of the work that come after the checklist is filled in.
Once you have written down who, what, solution, success signal, and failure signal, AI can help you draft the landing-page copy, generate the survey questions, scaffold the analytics, summarize the interview transcripts, and write the read-out. The execution work compresses by an order of magnitude.
AI is not great at filling the checklist in for you. It does not know your customer. It does not know what problem they actually have versus what problem you wish they had. It will cheerfully suggest a customer segment, a problem statement, a success metric — all of them plausible, none of them grounded in anyone you have talked to. The checklist is the part that requires you to know things about humans, which AI does not.
The second edition of The Real Startup Book annotates each chapter with where AI helps and where it gets you killed. For the hypothesis checklist the annotation is short: AI does not help. Write it down yourself, by hand if you have to, and only then ask AI to help you execute.
Read the chapter
The hypothesis checklist chapter in The Real Startup Book second edition walks through each of the five questions with worked examples — including a few from the AI era where the founder filled in the checklist correctly and the experiment saved them six months, and a couple where the founder skipped it and the experiment cost them a year.
Next post in the series: the smoke test. Same experiment we have been running since 2015, now an order of magnitude cheaper to run badly.
— Tristan
Comments
Loading comments…
Leave a comment