Value Proposition Test - Fake Door
Fake Door Smoke Test
In Brief
People are bad at predicting what they want, but they are great at reacting to new offers. Building just the entry to an offer makes it possible to predict future reactions based on actual behavior. This test is also known as a 404 page test.
Helps Answer
- Is the customer interested enough to click?
- Which target audience is attracted?
- Which words, icons, or images work well?
Tags
- Testing
- Quantitative validation
- Positioning
- Value proposition testing
- Feature testing
Description
The fake door or 404 page test is a quantitative validation method that is useful in two circumstances:
- To test if your idea sparks the interest of potential customers.
- To test if a new feature is a welcome addition to an existing product.
The idea of this test is to build a fake advertisement or button on a website to gauge the interest of potential users by looking at the clickthrough rate. The fake door could link to a “coming soon” page and prompt the user to add their email address to be notified when the product or feature is available.
Advantages:
An idea can be quickly validated without spending a lot of time and money on development. Data is generated on users' actual behavior rather than asking them if they would be interested.
Ethics:
Fake door testing can be thought of as tricking customers and decreasing the credibility of the product or business. When used carefully and in moderation, it can save time and money and lead to the development of a better product.
Clear communication that the feature or product is in development can prevent the user becoming upset that the product or feature isn’t available.
Internal and External Fake Doors:
The internal fake door is great for testing if a potential new feature appeals to current users. Usually a link or button is created on a website for this new function.
One example would be a button on a local news website that says “Switch to map view” when the information is currently presented in a tile view. If users are satisfied with how the information is presented, it is likely they will not click this button. If users assume a map view would better suit their needs, it is likely they will click. The clickthrough rate (the amount of clicks divided by the number of views) will help to gauge user interest.
Be careful when interpreting results. For example, the location of a button can have more influence than the button itself. Always experiment with placing buttons in different locations.
Using an external fake door is a great test to see if your concept is interesting for a certain audience. A common way to test this is using a "fake ad." For example, say you have the idea to start an online community for Kingfisher enthusiasts. Before building a complete website, you set up a test and check if there are enough Kingfisher enthusiasts looking for an online community. You determine beforehand that if more than five percent of all views convert to a click, enough users are interested. This allows you to simply test if your idea suits the need.
Be careful. The text in the header and body of your ad have a great impact on how people react to the advertisement. Always try a variety of sentences in your headers and body of the ad to overcome bias.
Fake doors are not suitable for testing your minimum viable product and essential functionalities, such as your registration process.
Time Commitment and Resources
- 1 hour to formulate a hypothesis
- 1-4 hours to create the fake doors, depending on the type
- 1-10 days to generate enough traffic for a representative sample size (depending on the amount of traffic or money you’re willing to spend)
- 1 hour to evaluate and write down learnings
How to
- Write down the hypothesis. Each test starts with writing down your key assumptions in the form of a hypothesis.
- Build the fake door(s). Make sure you represent the feature the same way it would be built. If you exaggerate the button or particularly highlight the link, your results will be biased and therefore useless.
- Make sure that you also include a method of measurement. With fake ads the channel takes care of this: Google, Facebook, and Twitter all give metrics on how your ads perform. And you might as well track the number of visits on the page you send them to. If the link is on your website, add a "Google tracking event" to your button, or create a special landing page so you can track the number of visitors.
- Measure.
- Run the experiment. And try to be patient. Don’t draw conclusions too soon; wait until the test is complete (in this case when you have at least 1000 visits or 10,000 views).
Interpreting Results
Check your initial hypothesis and see if the results match your success rate. Depending on your results, you have three options:
- Success! Congratulations, you were right. You decide to build the feature.
- No success! You pivot. Try reframing the link or button. A different word/icon/picture might yield a totally different outcome.
- No success! Congratulations. You prevented yourself from wasting scarce resources.
Potential Biases
- Copy bias: The words used in the fake door will greatly influence your results. Make sure to try copy variations.
- Location bias. The location of your ad (which platform) or your link (relative position on your website) will also influence your results. Make sure you try different variations.
Field Tips
- Don’t scare off your users. Testing is okay, but be careful not to overdo it.
- Making (too many) false promises by fake doors might give the user the impression of a lousy product or cause them to lose trust in your company.
- Make sure you have a significant number of users so you can accurately judge the results.
- Got a tip? Add a tweetable quote by emailing us: [email protected]
Case Studies
- Buffer went from idea to paying customers in seven weeks
- Game studio Zynga has a strong reputation for using fake doors and other quantitative product tests. They test new game ideas by coming up with a five-word pitch for each one. Then they publish the pitch as a promotional link in their live games for a period of time to see how much interest it generates from existing users.
- Local Dutch information aggregator Tippiq.nl uses fake doors extensively in their proposition development, via a separate website with www.tippiqlabs.nl. All new ideas are tested first using fake ads that lead to a landing page on that website. On the landing page, the product/service is explained and the visitor is asked if they would like to use the product. After they have expressed their interest, they get a notification that the product is still in development and they can leave their email if they want to stay up-to-date.
- Got a case study? Add a link by emailing us: [email protected]
Tools
- Got a tool to recommend? Add a link by emailing us: [email protected]