Actionable Metrics vs. Fail Conditions: Kill Your Ideas First
Stop looking for flying penguins and start guarding against self-delusion.
By Tristan Kromer · Actionable metrics vs. Fail condition: Last week I wrote about the difference between an assumption and a hypothesis and Steven Diebold wrote a great response. Steve schooled me on a few topics and pointed out my lack of clarity in some areas. One thing he brought up deserves more debate: The Success/actionable Metrics. 
Bonus: I’m writing a more complete version of how to design great experiments as an open source “Real Book”, you can get on the download list here: Download
Declaring Victory!
A success metric is the idea that given any hypothesis, there is a metric which will indicate that, in the lean startup and experiment design jargon, the hypothesis is validated. Or simply put, it’s a good idea. Setting the success/actionable metrics seems easy. For example, our hypothesis might be:
The value proposition “Faster download speeds for your BitTorrent client” (Version A) will generate more sign ups than the value proposition “Conceal your IP address when downloading Game of Thrones with BitTorrent.” (Version B)
(Not the greatest hypothesis, but let’s roll with it for now.) 
The metric for measuring this hypothesis will be the % of unique visitors that sign up on version A vs. version B. If the conversion rate for B is 5%, then if the conversion rate for A is 25%, the hypothesis is considered validated. Victory! Now let’s look at some situations where it’s a bit harder to declare a clear victory.
Flying Penguins
If our hypothesis is, “some penguins can fly” we can very easily set a success metric that would prove this hypothesis. If we see at least one flying penguin (outside of the cinema), then clearly some penguins can fly. So we go look at 10 penguins in the zoo and…they can’t fly. But maybe these are the wrong kind of penguins. We can go to a different city, go to another zoo, and look at another 20 penguins. They still can’t fly. Maybe it’s only penguins in zoos that can’t fly. So we get a boat, go to Antartica, and look at 1000 penguins in the wild. They still can’t fly. But maybe they just don’t like to fly while people are watching! Clearly they wouldn’t have wings if they couldn’t fly, so we probably just haven’t found the flying ones…yet.
Quick Answer: Instead of setting success metrics for your experiments, set fail conditions — the specific threshold below which your idea is definitively dead. As entrepreneurs, we’re biased toward our vision and will rationalize any “success metric” miss (“19% is close enough to 20%!”). A fail condition, grounded in your business model’s minimum viability requirements, flips the framing from “what proves I’m right?” to “what proves I’m wrong?” — making it structurally harder to delude ourselves into chasing flying penguins.
The Slippery Slope of Failure
This is a quite common problem with startups:
Maybe these customers didn’t want to buy our product, but I’m sure if we keep looking we’ll find the ones that will.
If we define our actionable metrics at 20%, when the conversion comes in at 19%…it’s close enough. When the conversion is 15%…there’s room for improvement. When it’s 10%…clearly we need to spend more time optimizing. When it’s 5%…well…some people are still interested! When it’s 1%…maybe we’re not explaining it well enough. When it’s 0%…did we forget to install analytics? It’s almost impossible to accept failure. There’s always a potential rationalization. After all…we just haven’t succeeded…yet. Just like the penguins. 
The Scientific Method!
This general problem is well known to science… We can try over and over to disprove our hypothesis, until we have tried so often that we give up and accept the truth of the matter. That’s why we have the Theory of General Relativity instead of the Fact of General Relativity. Although the Theory of General Relativity allows us to launch rockets into space and lets our phones geolocate us, it’s still just a theory. Eventually, we may find some situation where the theory breaks down and won’t explain all the facts (e.g. quantum physics). Then we’ll have to come up with a new theory. So, instead of trying to prove a hypothesis with a actionable metrics, we should try to disprove the hypothesis with a Fail Condition.
Setting the Fail Condition
<img src=“/images/very-disappointed-customer-290x314.png” alt=""Why didn’t I spend more time on my experiment design?!"" class=“img-right” /> How many penguins do you need to observe before we are convinced that penguins can’t fly? 10? 50? 1000? The more penguins you look at, the higher your level of confidence in your conclusion that penguins can or can not fly. Science has clear criteria for what is an acceptable level of confidence (six-sigma), but we don’t have that luxury in entrepreneurship or in lean startup. Fortunately, we don’t need it. We don’t need to prove to everyone that penguins can’t fly. We just need to prove it to ourselves. Because ultimately, our goal is to build a business. If our business was to sell penguins cool flight goggles, we need to know that high wind speeds while flying is a serious problem for most penguins. If most penguins can’t fly, this is probably not a good business. So what % of penguins need to be able to fly for this business to be worth investing our time in? 50%? 30%? 1%? Focusing on just Adélie penguins, if we need a market of 2 million penguins to be able to make this a profitable business and there are about 3.75 million Adélie penguins, then we need almost 50% of any penguins we survey to be able to fly to make this business work. So how many do we need to look at? If we look at 10 and NONE can fly, then even with a margin of error ~7% due to a small sample size…this is a bad business.
Semantics
This is more than just semantics. Of course, a very smart and practiced individual might be able to set a success metric and be very rigorous when applying it.
19%? Nope…we set a Success Metric of 20%, let’s scrap this business.
Those are words no entrepreneur will ever utter. As entrepreneurs, we are biased towards our vision, towards optimizing, towards self delusion. The purpose of lean startup is to guard against this sort of cognitive bias. So
Key Takeaways
Bonus: I’m writing a more complete version of how to design great experiments as an open source “Real Book”, you can get on the download list here: Download Real Startup Book
Frequently Asked Questions
What are actionable metrics in lean startup experiments?
Actionable metrics are measurements tied directly to a specific hypothesis that help us determine whether an idea is working or not. Unlike vanity metrics, actionable metrics are connected to a clear threshold — for example, the conversion rate difference between two landing page variations. However, as product managers, we should be cautious about framing these purely as “success metrics” since that framing invites cognitive bias.
Why are success metrics dangerous for entrepreneurs?
As entrepreneurs, we are inherently biased toward our vision and prone to self-delusion. When we set a success metric of 20% and get 19%, we rationalize it as “close enough.” At 10%, we tell ourselves we just need to optimize more. This slippery slope means we almost never accept failure — there’s always a potential rationalization. A success metric gives us permission to keep moving the goalposts downward.
What is a fail condition and how is it different from a success metric?
A fail condition flips the framing: instead of asking “what proves we’re right?” we ask “what proves we’re wrong?” It’s rooted in the scientific method, where hypotheses are disproved rather than proved. By defining the specific result that would kill our idea — such as “if fewer than X% of penguins can fly, this business won’t work” — we create a harder threshold to rationalize away, guarding against the cognitive biases that plague startup founders.
How do you determine the right fail condition for an experiment?
Start with your business model requirements, not your hopes. Calculate the minimum threshold needed to make the business viable — for example, if you need a market of 2 million customers and the total addressable market is 3.75 million, roughly 50% must convert for the business to work. Then determine how large a sample you need to confidently detect whether you’re above or below that threshold. The fail condition is grounded in business reality, not optimism.
Can’t a disciplined entrepreneur just use success metrics rigorously?
In theory, yes — a very practiced individual could set a success metric and stick to it ruthlessly. But in practice, no entrepreneur will say “we got 19% instead of 20%, let’s scrap this business.” The entire purpose of lean startup methodology is to guard against cognitive bias. Framing our actionable metrics as fail conditions rather than success metrics structurally protects us from the rationalization trap that derails startups.
Comments
Loading comments…
Leave a comment