Comprehension Test Before Smoke Test: Stop Premature Surrender
Why a 0% conversion rate might mean your messaging failed, not your idea
By Tristan Kromer · 
I love smoke tests. They’re a great way of testing whether or not there is any serious demand for our value proposition. One common form of smoke test out of the Startup Real Book is a landing page test. The landing page states a value proposition with a call to action that asks the user to commit some form of value such as an email address or even money to sign up. Based on the % of users who signup, we get a rough signal from the market if the value proposition is in sufficient demand to build a minimal solution. If not, pivot! But as much as I like smoke tests, I hate premature surrender. That’s what happens when the signal we get from the market is NO, but we don’t truly grasp why. If you’re interested in cutting to the chase and reading the entire Real Book, you can download it here: Download
Quick Answer: Before running a landing page smoke test, we should first run a comprehension test — show our value proposition to ~20 people, then ask them to explain it back. If they can’t, a 0% conversion rate means our messaging failed, not our idea. This prevents premature surrender, where we pivot away from a viable concept simply because customers couldn’t understand the offer. As a bonus, participants who do comprehend it often suggest better language and keywords we can use to improve our smoke test.
Landing Page Test
Let’s say we’re selling a new type of shoe that cures plantar fasciitis. We put up a landing page test with our value prop and a “buy now” button. Then we put $1000 in to Google Adwords for “shoes” to drive traffic and sit back to wait for the money to start rolling in… 
…and our conversion rate is 0% Should we give up? That’s what the landing page test says. There is insufficient demand for this product. By this point, some of you are probably asking, “What the hell is plantar fasciitis?” Exactly. All those people coming to our site are asking the same thing. Is our test failing because customers aren’t interested? Or do they fail because they simply can’t understand the value proposition?
Comprehension Test
Illustration by Emily Chiu[/caption] Before we ever run a smoke test, we should run a comprehension test. It’s a simple test we can do that usually takes less than one hour:
- Write out our value proposition in 1-3 sentences.
- Show the value proposition to a participant for a few moments, just enough for them to read it. Then take it away.
- Ask them to explain the value proposition back to us in their own words.
If the participant’s explanation is roughly comparable to our own, we count that as a positive result. If not, then it’s a negative. For this sort of test, we generally want a sample size of about 20 people and a positive conversion of about 80%. Here’s a simple tool to track testing activities. The conversion has to be very high because regardless of what our value proposition is, people should understand it. Note: If you’re looking for some guidance on how to design lean experiments, download our Learn SMART Experiment Template.
Who?
Who we bring to this sort of test is important, but we don’t actually need to use our target customer. We just need someone with the same basic education level and vocabulary. That’s because we are not testing to see if someone wants our value prop, just if they can understand it. If our target market has a high school level education, then we can just walk out onto the street for participants. If our target market has a specialized vocabulary such as Chief Marketing Officers, we can probably run a comprehension test with anyone with a marketing background. MBAs seniors might do just fine and they are a lot more accessible than CMOs. Once we’re certain people understand our value proposition, only then can we run our smoke test and be confident about the results.
Qualitative vs. Quantitative
Don’t get caught up on qualitative vs. quantitative. Comprehension tests are both. We get a yes/no quantitative response depending on the % of people that can successfully explain our value proposition. We get the actual explanations as qualitative data. This data can be incredibly useful. If we use the comprehension test on our value prop “shoes that cure plantar fasciitis” we might get responses such as:
- “Don’t know.”
- “No idea.”
- “Don’t know.”
- “Something that cures a scary sounding disease.”
- “Don’t know.”
- “Shoes that help with a common runner’s condition that causes foot pain in the morning.”
Clearly the first five aren’t very useful, but the last one is. The last response is from someone who actually understood, probably a runner. Plantar fasciitis is a very common problem for runners. Although our original value proposition failed the comprehension test, our respondent has given us a new value proposition to test! Not only that, they’ve suggested some great keywords we can use for our Adwords campaign…something a lot more specific than “shoes” and with a lot more search volume than “shoes for plantar fasciitis.”
Rules of Thumb
- Before testing commitment, test comprehension
- Qualitative vs. quantitative is a useless distinction, we need both.
If you’re interested in reading the entire Real Book, you can download it here: Download
Frequently Asked Questions
What is a comprehension test in lean startup?
A comprehension test checks whether people can understand our value proposition before we test demand. We show our value proposition to about 20 participants, let them read it briefly, then ask them to explain it back in their own words. We’re looking for roughly 80% of participants to accurately restate the proposition, confirming the message is clear enough to proceed with smoke tests.
Why does my landing page smoke test get zero conversions?
A 0% conversion rate doesn’t necessarily mean there’s no demand — it may mean people can’t understand what we’re offering. As product managers, we need to distinguish between lack of interest and lack of comprehension. Running a comprehension test before a smoke test helps us determine whether the value proposition itself is unclear, so we don’t prematurely abandon a viable idea.
Do I need my target customer for a comprehension test?
No. We just need someone with the same basic education level and vocabulary as our target market. Since we’re testing understanding rather than desire, we can use more accessible substitutes. For example, if we’re targeting Chief Marketing Officers, MBA seniors with a marketing background would work fine and are much easier to recruit.
Is a comprehension test qualitative or quantitative?
It’s both. We get quantitative data from the pass/fail percentage of participants who correctly explain our value proposition. We also get qualitative data from their actual explanations, which can be incredibly useful. Participants who do understand the proposition may reframe it in language that suggests better messaging, new keywords for ad campaigns, or even an entirely new value proposition to test.
What order should I run lean startup experiments in?
We should always test comprehension before testing commitment. If we skip straight to a smoke test like a landing page, a negative result is ambiguous — we won’t know if customers rejected the offer or simply couldn’t understand it. Confirming comprehension first ensures that when we do run a smoke test, the signal we get about demand is reliable and actionable.
Comments
Loading comments…
Leave a comment