In 2009 I saw Kent Beck speak for the first time at the very first Lean Startup Conference. Back then it was called the “Startup Lessons Learned” Conference. His topic? Build Measure Learn….and goats.
Kent was going on and on about an itchy goat. I had no idea who he was or why he was talking about this goat or why he was scratching it on the back. It was weird and confusing. Even Eric Ries looked a bit baffled standing on the sidelines. But it was brilliant. Kent said something that stuck with me.
The Build Measure Learn loop is backwards.
But let’s start with some background. (If you’re already very familiar with lean startup, you can skip the background.)
What is the Build Measure Learn Loop?
The Build-Measure-Learn (BML) loop is the most fundamental concept of the Lean Startup. It represents the basic process of rapid iteration based on direct consumer feedback on functional products. This rapid iteration validates or invalidates basic assumptions about the market need and whether the product satisfied that need.
The basic concept is similar to a number of loops found in other domains such as:
- Think-Make-Check in Lean UX
- Observe-Orient-Decide-Act (OODA) used in the US Air Force
- Plan-Do-Check-Act (PDCA) in Quality Control
- Assessment–Diagnosis–Planning–Implementation–Evaluation (ADPIE) in nursing
The BML loop isn’t really anything new. Every two year old is familiar with “If at first you don’t succeed, try, try again.”
What’s new about Build-Measure-Learn?
The most common differences emphasized by advocates of lean startup are:
- Business focus
Lean startup emphasizes speed above all else.
Partly, that’s a result of being created in the technology sector by a software engineer. Silicon Valley is famous for emphasizing rapid iteration to get to market faster.
24 hour caffeine fueled hackathons are the norm. Participants are expected to showcase working software at the end. Prizes or just social approval, are the reward. Just watch Mark Zuckerberg (played by Jesse Eisenberg in The Social Network) to get the idea.
There are other industries where one can create a product in 24 hours. Personally, I’m a mediocre cook, but I could prototype a working restaurant menu in 24 hours. But there is no other industry where a product could be ready to scale to tens of thousands of users almost instantaneously.
In lean startup, measuring has a strong emphasis on quantitative data.
The rapid scale of the tech industry and ability to measure detailed user behavior led to more data than anyone had ever had. The amount of data and detail available on a website is like Elf on the Shelf on overdrive. Every click, every mouse move is being tracked whether or not cookies are enabled.
That emphasis has mellowed over the years as lean startup has overlapped and merged with design thinking and user research techniques. But many lean startup evangelists still have a strong bias towards quantitative data above all else.
Lastly, lean startup is not about testing code or a product. That rapid iteration isn’t meant to determine if the product is working.
The Build Measure Learn loop is about testing a Business Model.
Lean Startup comes from agile software development and shares many of its principles. While agile evangelists will advocate many of the exact same tactics as lean startup, the primary focus is on building software.
Lean Startup explicitly focuses on testing a complete business model. That means simply producing working software is not a complete iteration. That working software needs to interact with customers. So the data doesn’t relate to error or defect rate, but relates to whether the business model is viable. (Hence the V in relative concept of Minimum Viable Product.)
Now back to Kent Beck…
Kent Beck’s Itchy Goat
The story Kent told involved baby goats, adult goats, and a chainsaw. I won’t get into but no goats were harmed during the story. It was a story about figuring out how to make his goat happy.
Personally, I have no idea how to tell if a goat is happy. To clarify, yes. Kent Beck owns goats. Yes, it was a strained metaphor. No, I’m not going to explain the story more than that. It was hard to capture and no one other than Kent Beck could have pulled it off. Don’t overthink it.
Bottom line is that through trial and error, he found the itchy spot. He suddenly received an “outsized reaction” for an activity he had been repeating in a number of different places to no effect.
But then Kent said, ”the Build Measure Learn loop is backwards.”
What’s wrong with the Build-Measure-Learn?
The BML loop presupposes that if we start building something and slap some analytics on it then we will inevitably learn something.
That means building a Minimum Viable Product.
So we hack something fun together over the weekend, release it, see what happens. Then we can decide if we want to continue.
That’s certainly one approach. Let’s just throw it out there and see if it works!
How will we know if it works?
Slap google analytics on it and throw up a post on Hacker News or Reddit. The post will be voted to the top in meritocratic fashion, and then we’ll see if anyone sign up.
Here’s the problem:
- Getting to #1 on Hacker News? – A vanity metric.
- # of sign ups? – A vanity metric.
There are over 7 billion people on the planet. If someone is willing to buy the shake weight, someone will sign up for whatever idiotic startup idea we come up with. Even if it’s just as a joke.
But getting curiosity sign ups is insufficient.
Who is that person that signed up? How many sign ups do we need for success? Why do they want it? How many of them are there? Are they willing to pay for it?
If we’re building a product to target soccer moms, then a posting on Hacker News has no bearing on our product. If anything, it’s going to give us a false negative because we’re talking to the wrong people.
We can not measure our sign up conversion rate without having a clear target market.
(Note: “Everyone” is not a clear target market. More on that in another post.)
If Build Measure Learn is just trial and error, then it’s no better than hope. This is what is affectionately called the “Field of Dreams” approach by some and “Spaghetti testing” by others.
Build it, and they will come!
Build Measure Learn Backwards
To learn quickly in any domain, we have to focus and have a clear plan of how and what we want to learn.
If we want to play the guitar, just picking it up and strumming isn’t going to get us very far very fast. To learn quickly, we need to break it down to simple elements and practice. Melody and harmony. Different chords and scales. Left hand and right hand.
There’s a clear goal right at the beginning of each practice session to focus on one specific element and master it.
Same goes for the Lean Startup. To learn quickly, we reverse the BML loop.
Step 1 – Identify the Learning Goal
The first thing we need to do is decide what to learn. Scientists don’t just start throwing chemicals into a vat and randomly feed it to babies to see what happens. They didn’t build the Large Hadron Collider for the hell of it.
A good scientist forms a hypothesis, then carefully designs an experiment with a control group to measure the effect of that experiment. Only then do they implement the experiment and learn something.
So start the loop by figuring out what we’re trying to learn. (Our team at Kromatic calls this the >Learning Goal>.)
This goal is best phrased as a question. Don’t say, “I want to learn about product pricing.” Say, “What price will our customers pay for our product?”
Step 2 – Specify the Data
Rephrasing the learning goal as a question is just a cognitive trick. Thinking about pricing in the abstract seems hard. Especially if we’ve never done it before.
Framing our learning goal as a question makes it easier to figure out what sort of data would help us answer this question. We need to know what to measure.
Our question is, “What price will our customer pay for our product?” Based on that, there are some clear pieces of data that will help us:
- % of people who actually pay $10, $20, or $9.99 for the product
- The average price paid for a comparable product
- The qualitative degree of the customer’s need for a solution
Each of these potential data points is something we can now go and get.
Step 3 – Select a Method
This is not where we go and >build> a Minimum Viable Product. We should build a Minimum Viable Test.
(Parker Thompson also proposed the term Minimum Viable Interaction which I also like tremendously.)
We know what data we need, so we should go out and get it with the least amount of effort possible.
To get the “% of people who actually pay $10, $20, or $9.99 for the product,” we don’t necessarily need to build a product.
We could try just building a landing page. Or we could even just attempt to sell the product face to face. We don’t even need the real product, a prototype or even a fancy box that looks like the real product would suffice.
To find “the average price paid for a comparable product” then we don’t need to build anything. We should just go shopping.
If we need “the qualitative degree of the customer’s need for a solution” then we should simply go talk to customers.
The qualitative degree of the customer’s need for a solution
So if we really want to start with Build, let’s do it. I personally enjoy building random toys from time to time just for the fun of it. But we shouldn’t ourselves about what we’re learning.
If we really want to learn about our business, we need to start by figuring out what we want to learn.
- Identify the Learning Goal
- Specify the Data
- Select a Method
Gain confidence that you're running the right kind of tests in a five-week series of live sessions and online exercises with our Running Better Experiments program. Refine your experiment process to reduce bias, uncover actionable results, and define clear next steps.
Click here to learn more →