2.4 Generative vs. Evaluative
Is our question open-ended with many possible answers or does it have a single true answer?
Similar to market vs. product, we can separate our questions into those that require generating ideas vs. those that provide factual answers.
Generative
- Who is our customer?
- What are their pains?
- What job needs to be done?
- How can we solve this problem?
- How do we find our customers?
More generative questions
- How are they doing this job today?
- Is our customer segment too broad?
- How much will this customer segment pay?
- How do we convince this customer segment to buy?
Evaluative
- Will customers pay $9.99?
- Should we target segment A or B?
- Will this feature increase our sales?
- Will more than 10% of users engage?
- Does a free trial increase paid conversions?
More evaluative questions
- What is the cost of acquiring a customer?
- Will a landing page targeted at this segment outperform a wider audience?
Telling them apart
The easiest way to tell if you have a generative question is if it is open-ended or not. If the answer to the question could be a list of possible ideas, it’s generative.
If the answer to the question is closed-ended, it’s probably evaluative. Yes/no, multiple choice, and questions about a specific fact are strong candidates. Quantitative questions are likely evaluative questions where there is one clear factual answer calculated from the data.
However, in some cases a question can appear to be closed-ended but have an ambiguous, poorly defined hypothesis backing it. To that end, the next chapter will provide some best practices around defining hypotheses.
Where Should We Start?
As with market vs. product, this book is semi-agnostic about where we start. However, unless we can tie a clear and well defined hypothesis to an evaluative question, we are almost always better off starting with generative methods.
