Product Market Fit Survey: Why the 40% Test Gives False Positives
Sean Ellis' test said we had PMF. Our startup said otherwise.
By Tristan Kromer · · 5 min read As I’ve mentioned before, being a lean startup is hard. Beyond the psychological pressures of trying desperately to fail at a rapid rate, there are a million ways to test a hypothesis, and a million hypotheses to test. Finding Product / Market fit is hard. Testing it is harder. How do we know which type of test to use, and when to use it?
- Should we do a smoke test?
- A concierge test?
- Talk to our customers?
All we want is to build something people want. Which one of these tests will actually tell us if we have the fabled Product / Market Fit? 
(This is a giant topic that I’m going to explore over the next several posts. You can cut to the chase by downloading the Product / Market Fit Storyboard.) Let’s start with the obvious…
What is Product / Market Fit?
Product/Market Fit was a phrase coined by Marc Andreessen. As Kali Albright notes in his answer on Quora, Marc said,
Product/market fit means being in a good market with a product that can satisfy that market.
For context, Marc is a venture capitalist. So a “good market” is one that is sufficiently large and growing at a rate to warrant a VC being even remotely interested.
Scaling marketing
Sean Ellis, who is often credited with popularizing the term, had a similar context. He said that he was only able to effectively scale marketing after the company had achieved PMF and he used his Customer Development Survey survey to measure their readiness. Product/Market Fit was the foundation of his Startup Marketing Pyramid. So Product/Market Fit is about scaling. Put simply, it’s when you stop wondering if your product actually solves a problem and you start wondering how you’re going to deal with all the people trying to buy your product and how to get even more of them to your storefront.
Quick Answer: The Sean Ellis product market fit survey (asking users “How would you feel if you could no longer use this product?” and targeting 40%+ “very disappointed”) is a useful but incomplete test. Scoring below 40% likely means you lack PMF, but scoring above it doesn’t guarantee you have it — early-stage products often get false positives because customers respond to the promise of a solution, not the actual product. As product managers, we need a roadmap of multiple experiments, not a single survey, to truly validate PMF.
How do I know if I have Product / Market Fit?
Sean used his Customer Development Survey to measure Product/Market Fit, stating that a 40% or higher result was necessary to have Product / Market Fit:
How would you feel if you could no longer use ?
- Very disappointed
- Somewhat disappointed
- Not disappointed (it isn’t really that useful)
- N/A – I no longer use
He went on to say:
If you find that over 40% of your users are saying that they would be “very disappointed” without your product, there is a great chance you can build sustainable, scalable customer acquisition growth on this “must have” product… Those that were above 40% are generally able to sustainably scale the businesses; those significantly below 40% always seem to struggle.
Asked about this at the March 2010 Lean Startup Circle meetup in San Francisco, Sean repeated that the >40% result was necessary, but he added that it was not enough by itself — a detail I can sympathize with.
StartupSquare
My first startup in Silicon Valley, StartupSquare, had >40% on the Customer Development Survey, and unequivocally did not have PMF. We had Problem / Solution Fit, but not Product / Market Fit. Our product simply didn’t work that well. In fact, it didn’t do too much of anything. Instead, we had measured our customer’s great burning desire for a solution, but had not succeeded in creating that solution. Furthermore, the market was too small at that time to merit additional investment in creating a better solution. Any additional time or money spent would have been wasted on a fundamentally broken business model. It failed both Marc Andreessen and Sean Ellis’ definitions.
Jargon
So here is my more specific and overly wordy definition:
Product/Market Fit is sufficient demand in a clearly defined marketplace for a product delivering a clearly defined value proposition to allow efficient (human or financial) capital expenditure to scale value creation.
If scaling the company results in increased costs without additional returns (either in revenue or, at the very least, company valuation) then the company is prematurely scaling prior to Product/Market Fit.
False positives
I suspect survey.io can give a false positive result when the product is early stage enough that customers are answering based on the promise of a solution instead of the actual product. (I admittedly have not, and will not, do the more rigorous academic research to prove this conclusively. But I only need one example to show that a false positive is possible.) So while scoring <40% on Sean’s survey probably indicates that you do not have PMF, >40% does not necessarily mean that you have Product / Market Fit.
Product / Market Fit and business models
Over the next few posts, I’m going to delve into this topic and how I use Product / Market Fit Storyboarding to create a roadmap of experiments to prove various hypotheses driving towards PMF. (If you’d like to cut to the chase, you can download the Product / Market Fit Storyboard.) (Illustration credit for the good ones goes to Emily Chiu.)
Frequently Asked Questions
What is a product market fit survey and how does it work?
A product market fit survey, often called the Sean Ellis test or Customer Development Survey, asks users: “How would you feel if you could no longer use [product]?” If over 40% respond “very disappointed,” it suggests you may have product/market fit. However, as we’ve seen, this threshold is necessary but not sufficient on its own to confirm PMF.
Can the Sean Ellis 40% test give false positives?
Yes. A product market fit survey can give false positive results, especially with early-stage products where customers respond based on the promise of a solution rather than the actual product experience. StartupSquare scored above 40% yet clearly lacked PMF — the product didn’t work well enough, and the market was too small. So while scoring below 40% likely means you don’t have PMF, scoring above it doesn’t guarantee you do.
What’s the difference between problem/solution fit and product/market fit?
Problem/solution fit means customers have a burning desire for a solution to their problem — but it doesn’t mean your product actually delivers that solution effectively. Product/market fit goes further: it requires sufficient demand in a clearly defined market for a product that delivers real value, enabling you to scale efficiently. As product managers, we need to validate both stages separately.
How do you actually define product/market fit?
Product/market fit is sufficient demand in a clearly defined marketplace for a product delivering a clearly defined value proposition, allowing efficient capital expenditure to scale value creation. Put simply, it’s when you stop questioning whether your product solves a real problem and start figuring out how to handle all the people trying to buy it — and how to reach even more of them.
What tests should I use to validate product/market fit beyond surveys?
No single test — whether a smoke test, concierge test, customer interview, or product market fit survey — will definitively prove PMF on its own. We need a roadmap of experiments that test different hypotheses across demand, solution quality, and market size. Tools like Product/Market Fit Storyboarding can help map out which experiments to run and in what order to systematically build confidence in PMF.
Comments
Loading comments…
Leave a comment