False Positives and Product Market Fit Survey – How do we test for it?
As I’ve mentioned, being a lean startup is hard. Beyond the psychological pressures of trying desperately to fail at a rapid rate, there are a million ways to test a hypothesis, and a million hypotheses. Finding product market fit is hard. And testing it is harder.
How do we know which type of test to use, and when to use it?
- Should we do a smoke test?
- A concierge test?
- Talk to our customers?
All we want is to build something people want. Which one of these tests will actually tell us if we have the fabled Product / Market Fit?
(This is a giant topic that I’m going to explore over the next several posts. You can cut to the chase by downloading the Product / Market Fit Storyboard here. If you’re in San Francisco, you can also join the Your MVP Sucks! Workshop.)
Let’s start with the obvious…
What is Product / Market Fit?
Product/Market Fit was a phrase coined by Marc Andreessen. As Kali Albright notes in his answer on Quora, Marc said,
Product/market fit means being in a good market with a product that can satisfy that market.
For context, Marc is a venture capitalist. So a “good market” is one that is sufficiently large and growing at a rate to warrant a VC being even remotely interested.
Sean Ellis, who is often credited with popularizing the term, had a similar context. He said that he was only able to effectively scale marketing after the company had achieved PMF and he used his Customer Development Survey survey to measure their readiness. Product/Market Fit was the foundation of his Startup Marketing Pyramid.
So, Product/Market Fit is about scaling.
Put simply, it’s when you stop wondering if your product actually solves a problem and you start wondering how you’re going to deal with all the people trying to buy your product and how to get even more of them to your storefront.
How do I know if I have Product / Market Fit?
How would you feel if you could no longer use [product]?
- Very disappointed
- Somewhat disappointed
- Not disappointed (it isn’t really that useful)
- N/A – I no longer use [product]
He went on to say:
If you find that over 40% of your users are saying that they would be “very disappointed” without your product, there is a great chance you can build sustainable, scalable customer acquisition growth on this “must have” product… Those that were above 40% are generally able to sustainably scale the businesses; those significantly below 40% always seem to struggle.
Asked specifically about this at the March 2010 Lean Startup Circle meetup in San Francisco, Sean repeated that the >40% result was necessary but not sufficient and I know the failings of a misinterpreted survey first hand.
My first startup in Silicon Valley, StartupSquare, had >40% on the Customer Development Survey and unequivocally did not have PMF. We had Problem / Solution Fit, but not Product / Market Fit.
Our product simply didn’t work that well. In fact, it didn’t do too much of anything.
Instead, we had measured our customer’s great burning desire for a solution, but had not succeeded in creating that solution.
Furthermore, the market was too small at that time to merit additional investment in creating a better solution. Any additional time or money spent would have been wasted on a fundamentally broken business model.
It failed both Marc Andreessen and Sean Ellis’ definitions.
So here is my more specific and overly wordy definition:
Product/Market Fit is sufficient demand in a clearly defined marketplace for a product delivering a clearly defined value proposition to allow efficient (human or financial) capital expenditure to scale value creation.
If scaling the company results in increased costs without additional returns (either in revenue or, at the very least, company valuation) then the company is prematurely scaling prior to Product/Market Fit.
I suspect survey.io can give a false positive result when the product is early stage enough that customers are answering based on the promise of a solution instead of the actual product.
(I admittedly have not, and will not, do the more rigorous academic research to prove this conclusively. But I only need one example to show that a false positive is possible.)40% very disappointed on survey.io is necessary but not sufficient to have Product/Market Fit Click To Tweet
So while having <40% on Sean’s survey probably does indicate that you do not have PMF, >40% is a very very strong indicator, but does not necessarily mean with certainty that you have product/market fit.
Product / Market Fit and Business ModelsA single survey will never tell you if you have Product/Market Fit Click To Tweet
Over the next few posts, I’m going to delve into this topic and how I use Product/Market Fit Storyboarding to create a roadmap of experiments to prove various hypotheses driving towards Product/Market Fit.
(Illustration credit for the good ones goes to Emily Chiu)