Generative vs. Evaluative

A 2x2 framework matrix mapping market vs. product on the columns and generative research vs. evaluative experiment on the rows, with iconographic cell content for each quadrant

Is our question open-ended with many possible answers or does it have a single true answer?

Similar to market vs. product, we can separate our questions into those that require generating ideas vs. those that provide factual answers.

Generative

Who is our customer?
What are their pains?
What job needs to be done?
How can we solve this problem?
How do we find our customers?

Evaluative

Will this specific customer segment pay $9.99 for a solution?
Should we target segment A or segment B?
Will this specific feature increase our sales?
Will more than 10% of users engage with this feature in their first session?
Does offering a free trial increase paid conversions versus immediate purchase?

Telling them apart

The easiest way to tell if you have a generative question is whether or not it is open-ended. If the answer to the question could be a list of possible ideas, it’s generative.

If the answer to the question is closed-ended, it’s probably evaluative. Yes/no, multiple choice, questions about a specific fact are strong candidates. Quantitative questions in general are likely evaluative questions where there is one clear factual answer calculated from the data.

However, in some cases a question can appear to be closed-ended but have an ambiguous, poorly defined hypothesis backing it. To that end, the next chapter will provide some best practices around defining hypotheses.

Where Should We Start?

As with market vs. product, this book is agnostic about where we start. However, unless we can tie a clear and well defined hypothesis to an evaluative question, we are almost always better off starting with generative methods.