Hypothesis Checklist

A figure peering intently through a magnifying glass to examine assumptions

Writing a Good Hypothesis

A sure way to keep an experiment on track is to start with a strong, clear hypothesis — one that tells us what we are testing, what we expect, and when the experiment has succeeded or failed.

If we mistake a bad hypothesis for a good one, we might choose to run an evaluative experiment when we really need generative research. That can lead to wasted time and misunderstood data.

“Our customers really want our product.”

This hypothesis is bad for a number of reasons. The most obvious is that it’s tautologically correct. If they are already our customers, then they’ve already proven they want our product. So this hypothesis isn’t worth investigating further.

This type of flawed hypothesis is common. We often inject hypotheses that seem profound but really just express things we already know are true. Here is a more subtle example:

“If visitors see our landing page, then they will sign up for our service.”

While not as flawed as the first example, it has fundamental problems that would prevent us from designing a good experiment. A badly designed experiment will most likely provide ambiguous data that we can’t interpret without bias.

In this hypothesis, several things are unclear:

Which visitors? Men? Women? Under 30? Over 30? Mobile or desktop users?
What does signing up require? Just an email? Or payment?
What percentage of visitors signing up counts as success?
What images and design will we use for our landing page?

Without defining the hypothesis very clearly, we might also argue about the results. Is 5% a good conversion rate? Did they not sign up because the price was too high or because the value proposition was confusing?

When we do not have a clear, well-defined, and falsifiable hypothesis, we are better served by doing generative research instead of conducting an experiment. In this case, our learning goal could be “Which customer segment should we target?” or “What are the demographics and psychographics of our ideal customer profile?”

Given either of these goals, we are better off doing customer discovery interviews rather than testing our vague hypothesis. The outcome of the generative research should be a clear, well-defined, and falsifiable hypothesis that we can then test with an evaluative experiment.

Anatomy of a Hypothesis

A strong hypothesis has four elements:

The change — what we are going to change, launch, or create. It can be as simple as changing the color of a button or as big as launching a new marketing campaign. Only one thing should change at a time, otherwise there is no way to tell which aspect caused the effect.
The impact — the expected result. If we change x, then we expect y to happen.
The metric — a specific measurement that defines success or failure. A fail metric tells us when to scrap a project; a success metric tells us when to ship.
The timeframe — how long we will run the test. Too short and the data may be too thin; too long and we waste runway collecting unnecessary data.

There are many different formats for writing a hypothesis, but a good one should end up looking like this:

This new feature will cause a ten percent increase of new users visiting the homepage in three months.

(the change) — (the impact) — (the metric) — (the timeframe)

What Makes a Hypothesis Strong

Here is a short guide to the qualities a well-formed hypothesis should have.

Simple and Unambiguous

The hypothesis should be clear and unambiguous so that anyone reading it will understand the context and be able to clearly interpret the results.

“If 1000 Los Angeles skateboarders visit our webpage, our CTR will be at least 5%.”

While this is more specific, not everyone knows what CTR means, so we should avoid using any specialized vocabulary or jargon unless everyone on our team is familiar.

“If 1000 Los Angeles skateboarders visit our webpage, our click through rate to newsletter signup will be at least 5%.”

Measurable

“Our customers have a strong desire to donate to charitable causes.”

This hypothesis may be true, but it is not observable. At least not until we invent telepathy.

“Our customers donate to charitable causes twice per year.”

This new hypothesis could be more specific, but it is at least observable.

Describes a Relationship

“50 percent of students at Dalton High School get a C or lower in at least one class per year.”

This again may be true and it is observable, but it doesn’t tell us anything about the cause of the low grades. A good hypothesis should allow us to change one thing and observe the effect in another.

“Students at Dalton High School that study fewer than four hours a week get a C or lower in at least one class per year.”

There are other issues with this hypothesis, but at least it relates two or more variables to each other.

Cause and Effect

“During the summer, ice cream consumption increases and more people drown per day.”

This is a true statement, but it does not tell us how these two variables relate to one another. Are people drowning because they ate too much ice cream? Or are they eating more ice cream because they are sad about the drownings?

“During the summer, people who eat ice cream before swimming will drown at a higher rate than people who do not eat ice cream.”

This specifies a clear relationship and the causal direction of that relationship. Simply changing the sentence to an IF _______, THEN _______ structure can make the cause and effect relationship even more apparent:

“If we feed ice cream to people before they swim, then the average number of drownings per day will increase.”

Achievable

“If an astronaut in a stable orbit around a black hole extends one foot past the event horizon, then they will be pulled in entirely.”

There are many theoretical physicists who create hypotheses that are not testable now but may be testable in the future. As entrepreneurs, we should restrict our hypotheses to ones that can be tested in the immediate future and with our current resources. The timeframe should also be reasonable for the stage of the company — an experiment that extends past our runway is not achievable, no matter how well-formed.

Many things seem untestable today, but clever thinking can simplify the hypothesis into a testable first step.

Falsifiable

All of these conditions add up to a hypothesis being falsifiable. If a hypothesis cannot be proven incorrect, then it is not worth testing.

“There is an invisible, intangible tea cup floating between the Earth and Mars.”

When in doubt, ask: “What evidence would prove this hypothesis incorrect?”

If there is no amount of evidence that would prove our hypothesis invalid, then either the hypothesis is flawed or we are being very stubborn.

Common Mistakes

Too many variables. If we are testing multiple things at once, we cannot pinpoint which variable caused the results.
No achievable metric. Without a specific threshold, we have no clear point at which the experiment succeeds or fails.
Indirect success criteria. If the outcome could have been caused by any number of variables, we don’t know whether the experiment was responsible for the change.
Unrealistic timeframe. Some experiments take longer than others, but the timebox should be reasonable for the stage of the company and should not extend past our runway.

Other Frameworks

There are a number of frameworks and checklists for forming a hypothesis, one of which is popular enough to comment on to avoid confusion:

We believe this capability will result in this outcome and we will know we have succeeded when we see a measurable signal.

The entire sentence is not the hypothesis. Let’s break this into its parts:

We believe …

This section just confirms that we think the hypothesis is correct. It is not part of the hypothesis, and there are many situations where we may test a hypothesis that we believe is incorrect.

…<this capability> will result in <this outcome>…

That is the hypothesis.

…we will know we have succeeded when <we see a measurable signal>

That is the data we will collect, including any information about sample size, margin of error, success conditions, or fail criteria.

Other Resources

Got something to add? Share with the community.