7.1 A/B Testing

In Brief

Description

A/B testing shows two versions of the same element to your participants and lets their behavior decide which one wins. As the name suggests, you compare two versions (A and B) that are identical except for one change that might affect what participants do. Version A is usually the current version — the control, the baseline you measure against. Version B changes one thing — the variant. You split incoming participants randomly between the two and compare a single number for each: the share who take the action you care about, such as the percentage who sign up or buy.

The method is most often used on web pages, where the goal is to find changes that increase an outcome you care about: revenue, signups, downloads, leads, donations. Testing copy, layout, images, or colors can move those numbers, and even a small improvement on a high-traffic page can add up to a meaningful gain.

A purchase flow — the sequence of pages a visitor moves through from product page to completed order — is a common place to test, because reducing the share of people who drop out at each step turns directly into more sales. Whatever the metric, A/B testing measures whether a specific change actually moved it rather than leaving you to guess.

A/B testing changes one thing at a time. Testing several changes together (for example a new headline and a new button at once) is multivariate testing, a related but more involved method that needs more traffic to untangle which change did what. A/B testing also assumes you control the conditions and the random split; it does not apply to data you merely observe after the fact, such as survey responses or historical records, where you cannot assign people to versions.

Imagine a company, Acme Cables, that operates a web store selling cables. The company’s ultimate goal is to sell more cables and increase their yearly revenue, thus the checkout funnel is the first place Acme’s head of marketing will focus optimization efforts.

The “Buy” button on each product page is the first element visitors interact with at the start of the checkout process. The team hypothesizes that making the button more prominent on the page would lead to more clicks and therefore more purchases. The team then makes the button red in the variation and leaves the button grey in the original. They are able to quickly set up an A/B test using an A/B testing tool that pits the two variations against each other.

As the test runs, all visitors to the Acme Cables site are bucketed into a variation. They are equally divided between the red button page and the original page.

Once enough visitors have run through the test, the Acme team ends it and declares a winner. The results show that 4.5 percent of visitors clicked the red “Buy” button, against 1 percent for the original — a large enough gap, on enough visitors, to trust. Acme redesigns its product pages accordingly, and carries the lesson that red buttons convert better than grey on its site into later tests.

You can also test more than two versions of the same one thing — say three possible headlines. Splitting your participants (or, for an email, your recipients) into three groups instead of two is still the same method, and it is more efficient than running three separate two-way tests. Give a three-way test an extra couple of days so each group still gathers enough results to draw a conclusion.

Some AI experimentation platforms take this further, generating the versions themselves — copy, layout, even imagery — and automatically steering traffic toward whichever is winning instead of waiting for you to read a dashboard and pick. That suits high-volume optimization of a fixed goal and audience (pricing-page versions for a known segment, checkout tweaks, rotating ad creative), but it is the wrong tool for discovery: it optimizes hard against whatever metric you give it without questioning whether the metric or audience is right (see the local-optima and wrong-target pitfalls in Biases & Tips).

How to

Prep AI Prompt

Decide what you are testing, the metric that defines the win, and whether you have enough traffic before writing any variant code. AI tools can do the sample-size math and flag confounders for you, but the decisions are yours.

1. Confirm you have enough traffic. A/B testing only works once enough participants pass through each version that the result is unlikely to be chance — what statisticians call statistical significance. That takes volume. Use a sample-size calculator (see References) to check. As a rough guide, with under 1,000 unique visitors per week to the page, you likely cannot finish a meaningful test without waiting months. If that is you, switch to a method that learns from a handful of people — usability testing or customer discovery interviews — or test bolder changes that produce bigger, easier-to-detect differences. For B2B software teams, the best opportunity is usually inside the product (onboarding, feature adoption, the pricing page), not the marketing site: a feature flag (a switch that turns a change on for some users and off for others) lets you test against existing users where you control who sees what.
2. Define one hypothesis. State what you are testing, what metric will change, and by how much. Example: “Changing the onboarding flow from 5 steps to 3 will increase the activation rate by at least 5 percentage points.” One change at a time — testing several at once is multivariate testing, not A/B.
3. Pick the primary metric and the smallest difference worth detecting. Choose the single metric that matters most for this test — signups, activation, purchases, retention. Then decide the smallest improvement that would actually justify shipping the change, and write it down before you look at any data. This number (often called the minimum detectable effect) sets how big a sample you need and stops you from chasing a 0.1% win that isn’t worth the work.
4. Calculate the required sample size and duration. Feed your current baseline rate, the smallest difference worth detecting, and your daily traffic into a sample-size calculator. It tells you how many participants each version needs and how many days that will take. Decide now to run that long — do not stop early because the numbers “look significant” partway through (see Peeking bias in Biases & Tips).
5. Design the control and variant. Version A is the current experience (the control). Version B changes one thing only (the variant). Everything else stays identical. Write down what “shipping the change” means so the engineering work is scoped before the test, not after. Decide how many variants to run, too: two is the default, but a high-traffic site optimizing a fixed goal can use an AI-managed multi-variant platform to run many at once and auto-allocate traffic toward the winner.
6. Pilot the setup. Run the test on 1% of traffic for a day to confirm the tracking fires correctly, the split is even, and the metric records as expected. A broken tracking tag discovered on day 14 is a wasted test.

Analysis AI Prompt

1. Decide how sure you need to be — and don’t over-demand it. Statistical significance just measures how likely your result is real rather than a fluke of chance. Tools usually report it as a p-value: 0.05 means roughly a 5% chance you’d see a gap this big even if the two versions were truly identical (often stated as “95% confidence”). To check whether your numbers clear that bar — or how many more participants you’d need — run them through an experiment calculator, or hand the counts to a statistician or an AI to do the math. But 0.05 is a convention, not a requirement: an early-stage startup rarely has the traffic to reach it, and rarely needs to. Deciding whether to keep iterating on a headline is not the bet a public company makes when it changes checkout for millions of users. If the change is cheap and reversible and the early numbers point clearly one way, act on the weaker evidence and keep moving — holding out for a p-value your sample size will never reach is its own mistake. Match the bar to the cost of being wrong.
2. Check whether the result is worth acting on. A small p-value only tells you the difference is probably not zero — not that it is big enough to matter. A 0.1% improvement that is statistically solid on millions of participants may not be worth the work to build and maintain. Weigh the size of the change against what shipping it costs.
3. Segment the results. Aggregate results can hide that one segment loved the variant and another hated it. Slice by acquisition channel, device, new versus returning, and any segment where you expect different behavior. Treat segment effects as hypotheses for the next test, not conclusions.
4. Decide: ship, kill, or iterate. If the variant wins on both bars (statistical and practical), ship it and document the learning. If it loses, kill it and document why you thought it would work. If results are inconclusive, decide whether the next move is a bolder change, more traffic, or a different method — not a longer run of the same test.