A sure way to keep our sanity is to start with a strong hypothesis to give the experiment structure.
Lean startup practices turn project managers, business leaders, and designers into scientists who constantly validate their ideas by running an array of experiments. But experiments can get out of hand and turn perfectly sane people into mad scientists. A sure way to keep our sanity is to start with a strong hypothesis to give the experiment structure.
A strong hypothesis will tell us what we are testing and what we expect to get out of the test. By stating expectations, we are delineating the goals that the experiment has to hit to make it a success or failure. This will help us define when to determine that the experiment needs to be scrapped or the idea is ready to be taken to market.
Key elements of writing a good hypothesis:
A hypothesis should end up looking like this:
This new feature will cause a ten percent increase of new users visiting the homepage in three months.
(the change) ----------------------------(the metric)-------------------(the impact)----------------------(the timeframe)
Let’s break down each aspect:
This is the aspect that we are going to change, launch, or create that is going to affect our overall business or product. It can be as simple as changing the color of a button or as big as launching a new marketing campaign. Make sure that only one aspect is changed at a time, otherwise there is no way to tell which aspect contributed to the effect.
This is the expected results of the experiment. If we change x, then we expect y to happen.
This is a measurement that needs to be hit or surpassed. This can be a fail metric, where if the experiment does not meet the minimum goal, then the project must pivot to a completely new direction. This can also be a success metric, where the experiment is deemed a success if it hits the goal. Choosing between the success or fail metric is dependent on if we want the baseline to know when to scrap a project or when to launch a project.
This is the length of time it takes to run the test. If the timebox is too short, then the amount of data might be too small, or there might not have been enough time for effects to take place. But if the timebox is too long, we are wasting valuable time collecting unnecessary data.
Now let’s use these elements to form a hypothesis in an example scenario.
Say that you are a product manager at a startup that creates a mobile app to help waiters and waitresses keep track of their tips. You have noticed that users who document their tips four times a week have a higher retention rate. You want to see if you can increase the number of times current users use the app within a week.
What are you going to change within the app?
How about adding a notification system so the user can set reminders to ping them at the end of a shift?
What do you want the outcome to be?
You want more users to open the app four or more times in a week.
What is the metric of failure or success?
At this time you have 50,000 monthly users, and 10,000 use the app four days a week. You want to increase the current user’s rate of opening the app from 10,000 to 15,000. This translates to a ten percent increase.
How long are you going to run the test?
This always depends on a number of variables within the company, but let’s say that you are at a midsize company that has a little more time to get the correct data. So let’s say three months.
The end hypothesis would be:
If I add a notification feature that allows the waiter/waitress to set reminders to add in their tips, then I am going to see a ten percent increase in the number of users opening the app four times or more in a week over the next three months.
Now let's go through a worksheet that will test if you can figure out the strongest hypothesis for a given scenario. The answers are at the bottom.
Scenario A: You work for a company that rents out toddlers’ clothes. It is a monthly subscription where families get a box of five pieces of clothing, and when the toddler grows out of them, they return the clothes for a new box. The data shows that there might be a correlation between members who frequently send back items to higher customer retention rates. Your goal is to have members return more boxes. You have decided that you can do this by adding pieces that are seasonal, holiday themed, or super trendy so that the family will need to keep updating the clothes.
Scenario B: You already made your millions with the Uber for parrots, so you decided to invest your money into saving the manatees. You designed a tracking app that shows boaters where herds of manatees are sleeping so they don’t run the herds over. You are having a hard time getting the boaters to download the app, so you decide to start advertising. You want to conduct a test to see if a promotion will increase the app’s downloads.
Scenario C: Your labrador retriever is obsessed with a tennis ball and you are tired of throwing the slobbery thing. It inspired you to start a drone company that drops tennis balls and takes funny pictures of the dogs. Your customer-support team has received complaints that it is hard to understand how to download the pictures from the iPhone app. You want to test moving the photos section to various parts of the app.
Scenario D: You are so sick of wearing the same outfits that you developed an AI software to pick out your clothes every morning. A venture capitalist saw your tweets about it and gave you a million dollars to start the company. You need the AI to address weather conditions when choosing the clothes. You want to run an experiment to test a method for collecting data for when it is 80 degrees and sunny.
Scenario E: Men’s socks are a great way to jazz up an outfit, so you decided to start a men’s sock e-commerce store. Your customers are not completing the checkout process, and usability tests show that some users question the site’s security. You want to add a small adjustment to the payments page to see if more users complete the checkout process.
Learn from your mistakes. Look at the questions you got wrong and see which key element is either missing or vague.
|Is it simple and unambiguous?|
|I mean … is it simple...?|
|...and is it unambiguous?|
|Is it measurable?|
|Does it describe a relationship between two things?|
|Is the cause and effect relationship clear?|
|Is it achievable?|
|Is there any evidence that would convince us the hypothesis is invalid?|