Generative vs. Evaluative

Generative vs. Evaluative

Do we have a clear hypothesis to evaluate or do we need to generate a clear idea? This distinction depends on our understanding of what makes a clear hypothesis.

“Our customers really want our product.”

This hypothesis is bad for a number of reasons. The most obvious is that it’s tautologically correct — if they are already our customers, then they've already proven they want our product — which means it is not worth investigating further.

This type of flawed hypothesis is common. Here is a more subtle example:

“If 250 Los Angeles teachers were asked to treat minority students with more respect, then at least 50 teachers would do so.”

While not as flawed as the first example, it has fundamental problems that would prevent us from designing a good experiment. If we force an experiment, we will most likely have ambiguous data or be unable to correctly interpret it.

In this case, several things are unclear:

• Which teachers? Teachers at schools with a high percentage of minority students? What percentage is sufficient for this test?
• How would we ask the teachers? Would we ask each teacher differently? Would we let the principals ask them?
• What is respect in this context? Which behavior changes would indicate “more respect”?

Without defining the hypothesis very clearly, we might let the principals of schools ask the teachers on our behalf, and they might ask them with varying degrees of persuasiveness.

We might also argue about the results. Is calling a student “Mr.” or "Ms." instead of by their first name a sign of respect or a sign of sarcasm?

When we do not have a clear, well-defined, and falsifiable hypothesis, we are better served by doing generative research instead of conducting an experiment. In this case, our learning goal could be “Which teacher behaviors indicate teacher respect to minority students?”

Given this goal, we are better off doing customer discovery interviews (in this case, speaking to the students) rather that testing our vague hypothesis. The outcome of the generative research should be a clear, well-defined, and falsifiable hypothesis that we can then test with an evaluative experiment.

Defining good hypotheses can be a challenge, so here is a short checklist of points to consider.

Simple and Unambiguous

The hypothesis should be clear and unambiguous so that anyone reading it will understand the context and be able to clearly interpret the results.

“If 250 Los Angeles teachers were asked to treat minority students with more respect, then at least 50 teachers would do so.”

In this case, we may have different opinions on what “respect” means. For us to agree that someone is being treated with “more respect,” we must agree on which behaviors would indicate respect.

“If 250 Los Angeles teachers were asked to treat minority students with more respect, then at least 50 teachers would begin addressing their students with an honorific.”

While this is more specific, not everyone knows what an honorific is (in this case, Mr. or Ms.), so we should avoid using any specialized vocabulary or jargon.

“If 250 Los Angeles teachers were asked to treat minority students with more respect, then at least 50 teachers would begin addressing their students by ‘Mr./Ms.’ and their last name instead of their first name.”

Measurable

“Our customers have a strong desire to donate to charitable causes.”

This hypothesis may be true, but it is not observable. At least not until we invent telepathy.

“Our customers donate to charitable causes twice per year.”

This new hypothesis has some other issues, but it is at least observable.

Describes a Relationship

“50 percent of students at Dalton High School get a C or lower in at least one class per year.”

This again may be true and it is observable, but it doesn’t tell us anything about the cause of the low grades. A good hypothesis should allow us to change one thing and observe the effect in another.

“Students at Dalton High School that study fewer than four hours a week get a C or lower in at least one class per year.”

There are other issues with this hypothesis, but at least it relates two or more variables to each other.

Cause and Effect

“During the summer, ice cream consumption increases and more people drown per day.”

This is a true statement, but it does not tell us how these two variables relate to one another. Are people drowning because they ate too much ice cream? Or are they eating more ice cream because they are sad about the drownings?

“During the summer, people who eat ice cream before swimming will drown at a higher rate than people who do not eat ice cream.”

This specifies a clear relationship and the causal direction of that relationship. Simply changing the sentence to an IF _______, THEN _______ structure can make the cause and effect relationship even more apparent:

“If we feed ice cream to people before they swim, then the average number of drownings per day will increase.”

Achievable

“If an astronaut in a stable orbit around a black hole extends one foot past the event horizon of a black hole, then they will be pulled in entirely.”

There are many theoretical physicists who create a number of hypotheses that are not testable now but may be testable in the future. While this black hole/astronaut hypothesis is theoretically testable, it is not testable today.

Unfortunately, as entrepreneurs, we should restrict our hypotheses to ones that can be tested in the immediate future and with our current resources.

Many things seem untestable today, but clever application of lean thinking can simplify the hypothesis into a testable first step.

Falsifiable

All of these conditions add up to a hypothesis being falsifiable. If a hypothesis cannot be proven incorrect, then it is not relevant to run a test on it.

“There is an invisible, intangible tea cup floating between the Earth and Mars.”

When in doubt, we can ask ourselves, “What evidence would prove this hypothesis incorrect?”

If there is no amount of evidence that would prove our hypothesis is invalid, then either the hypothesis is flawed or we are very stubborn.

Other Frameworks

There are a number of frameworks and checklists for forming a hypothesis, one of which is popular enough to comment on to avoid confusion:

We believe this capability will result in this outcome and we will know we have succeeded when we see a measurable signal.

The entire sentence is not the hypothesis. Let’s break this into its parts:

We believe ...

This section just confirms that we think the hypothesis is correct. It is not part of the hypothesis, and there are many situations where we may test a hypothesis that we believe is incorrect.

...\ will result in \...

That is the hypothesis.

...we will know we have succeeded when \

That is the data we will collect, including any information about sample size, margin of error, success conditions, or fail criteria.