2.2 From Question to Method
Once we have the right questions, we need to find the right method. To do this, we need to understand the difference between Market vs. Product and Generative vs. Evaluative questions.
- Do we need to learn about the problem (ie, market) or the solution (ie, product)?
- Do we have a hypothesis to evaluate, or do we need to generate a clear idea?
Market questions relate to levels of customer demand, user problems, segmentation, or market size. Product questions relate to what solution would solve the market need.
Generative questions are open-ended where there is no single unique answer. Evaluative questions are specific, closed-ended questions where there is one specific, true answer such as yes or no.
The intersection of these two distinctions creates a 2x2 matrix with four quadrants:
Each of these quadrants relates to a set of questions and the methods that will generate the exact right data to answer those questions.
If we have a clear hypothesis of who our customer is and what we think they will pay for, then we can run an experiment from Evaluative Market Experiments, such as a Value Proposition Test.
If we don’t have a clear idea of who our customer is and it’s open-ended, we can do Generative Market Research, such as Data Mining.
Similarly, if we have a clear hypothesis of which features will solve the customer’s problems, we can run an experiment from Evaluative Product Experiments such as Wizard of Oz testing. If we do not know which features will lead to an acceptable solution, we can do Generative Product Research such as a Concierge Test to try to come up with new ideas.
Keep it Simple
This framework is deliberately simple. In reality, methods can deliver unexpected data and answer questions we haven’t even though of asking.
A single customer interview question (generative research) can generate a new hypothesis, and the next question can invalidate it. An MVP (minimum viable product) built to check a product feature (evaluative experiment) can surface new market hypotheses by accident.
However, this simple categorization helps keep us focused on our principle risks without distraction.
The Index of Questions and the Index of Methods show a list of example questions and their corresponding methods. But the next few chapters will go into more detail on how to distinguish the different types and help us navigate the four quadrants.
