80,000
visitors, split 50/50
Guardrails watched: refunds, errors, page speed
A/B testing compares two versions of a page, screen, email or flow by showing each to a random share of users and measuring which one does better on a metric chosen in advance. It replaces opinion with measured behavior, which makes it the Measure step of the lean startup build-measure-learn loop.
This guide covers what makes a test sound, how to write a hypothesis, which metrics to pick, how long to run a test, the main types of test, how to read the result and the mistakes that make results untrustworthy. For tools that split traffic and report results, see A/B testing tools.
Key takeaways
Test one clear hypothesis at a time, with a primary metric chosen before launch.
Most ideas do not win. A flat or losing result is information, not a failure.
Work out the sample size before you start and run full weeks. Stopping when the numbers look good is the most common way to fool yourself.
Guardrail metrics, such as refunds, errors and page speed, stop a short-term lift from hiding long-term harm.
With little traffic, test bigger changes higher in the funnel, or use interviews and usability tests instead.
What is A/B testing?
In an A/B test, the current version is the control (A) and the changed version is the variant (B). A testing tool assigns each visitor at random to one of them, so the two groups are alike in every way except the change. If B performs better on the chosen metric by more than chance would explain, the change probably caused it.
It is also called split testing. A test can have more than two versions (A/B/n), and a multivariate test changes several elements at once. The discipline is the same: a clear question, random assignment and a decision rule set before the data comes in.
Why test instead of deciding
Teams are often wrong about which ideas will work. In Harvard Business Review, Ron Kohavi and Stefan Thomke report that at Microsoft about one-third of tested ideas prove effective, one-third have neutral results and one-third have negative results. They also report that only about 10% to 20% of experiments at Google and Bing generate positive results.
Testing also limits risk. A new checkout layout can look better in a design review and still confuse buyers. Showing it to part of the traffic first means a bad idea costs a few days of data, not a quarter of revenue.
Parts of a sound test
A test needs more structure than 'try a new page and see':
An objective
The outcome that matters, such as completed purchases or qualified demo requests.
A hypothesis
The change, the reason it should help and the expected effect.
An audience and split
Who is included, and how traffic is divided between versions.
Reliable tracking
Conversion events checked before launch.
A decision rule
The metric, sample size and threshold that decide the result.
Writing a hypothesis
A useful format is: if we change X, then Y will happen, because Z. Each hypothesis has four parts:
Change
Add delivery and return details next to the add-to-cart button.
Reason
Support tickets show buyers hesitate over delivery times and returns.
Expected effect
More visitors add the item to their cart and complete the purchase.
Metric
Purchase completion rate, with add-to-cart rate as a supporting metric.
'Test shipping copy' is a task. The hypothesis above names a user concern and an outcome, so the result teaches something whether B wins or loses. Good hypotheses come from evidence: analytics, support tickets, session recordings, customer discovery interviews and past tests.
The A/B testing process
Treat testing as a repeatable workflow:
- 1
Find the problem. Use funnel reports, feedback and search data to see where users hesitate or drop off.
- 2
Write the hypothesis. State the change, the reason and the metric you expect to move.
- 3
Prioritize. Weigh likely impact, effort, traffic and risk against other ideas in the backlog.
- 4
Build the variant. Change only what the hypothesis needs, unless you are deliberately testing a full redesign.
- 5
Set up tracking and targeting. Confirm the audience, devices, conversion events and exclusions.
- 6
Run it to the planned sample size. Don't stop because early numbers look exciting.
- 7
Analyze. Review the primary metric, then supporting metrics, guardrails and segments.
- 8
Act and record. Roll out, iterate or drop the idea, and write down what you learned.
What to test first
Start where friction, business value and traffic overlap. Common candidates:
Headlines and value propositions
What the offer is, who it is for and why it matters.
Calls to action
Wording, placement and whether the ask comes too early.
Forms
Fewer fields, clearer labels, or a long form split into steps.
Product information
Sizing, delivery, compatibility and comparisons.
Navigation and filters
Helping people find the right item or plan faster.
Trust signals
Reviews, guarantees and return policies placed where doubts arise.
Page layout
Decision-making information moved to the moment it is needed.
Small changes can teach a lot when they answer a real concern. A button color test with no reason behind it usually teaches little.
Primary, supporting and guardrail metrics
Choose all three before launch:
| Metric | Role | Examples |
|---|---|---|
| Primary metric | Decides the result. Pick one | Purchase rate, signup completion, revenue per visitor |
| Supporting metrics | Explain why the result happened | Add-to-cart rate, size guide opens, review clicks |
| Guardrail metrics | Catch harm the primary metric misses | Refunds, support contacts, errors, page load time, unsubscribes |
Primary metric
Decides the result. Pick one
Examples: Purchase rate, signup completion, revenue per visitor
Supporting metrics
Explain why the result happened
Examples: Add-to-cart rate, size guide opens, review clicks
Guardrail metrics
Catch harm the primary metric misses
Examples: Refunds, support contacts, errors, page load time, unsubscribes
A variant that lifts button clicks but lowers completed purchases is not a win. Judging a test by whichever metric looks best afterward is one of the easiest ways to find a false winner.
Sample size and how long to run
Small differences need large samples. Work out the sample size with a calculator before launch, using your baseline rate and the smallest lift worth detecting.
Here is an illustrative case. A page converts at 4%, and the team wants to detect a lift to 4.4%, a 10% relative improvement. At the usual 95% confidence and 80% power, that needs about 39,500 visitors per version. A page with 2,000 visitors a week would need about 40 weeks, which is too long to be useful.
Run whole weeks, because behavior differs between weekdays and weekends. Avoid unusual periods such as a big sale unless that is what you want to learn about. And decide the end point in advance: checking every day and stopping at the first good-looking result inflates false positives.
Testing with little traffic
Many small sites and early products never reach the numbers above. Options that still produce learning:
- Test bigger changes: A new offer or page structure moves metrics more than a word change.
- Test higher in the funnel: Clicks and signups happen more often than purchases.
- Use qualitative methods: Five-user usability tests and interviews find the biggest problems quickly.
- Run fake-door or concierge tests: Measure demand before building. See the MVP guide.
- Call it what it is: A reasoned change based on research is fine, as long as nobody claims it was validated by a test.
Types of A/B test
The right type depends on what you change and who builds it:
| Type | Fits | Trade-off |
|---|---|---|
| Client-side | Copy, layout, images and forms changed in the browser | Fast to launch, but can slow pages and flicker |
| Server-side | Pricing, checkout logic, search results, onboarding | Cleaner and faster for users, but needs engineering |
| Multivariate | Several elements and their combinations at once | Needs much more traffic and is harder to read |
| Multi-page | A change across a journey, such as category to cart | Shows the full effect, but tracking must cover every step |
Client-side
Copy, layout, images and forms changed in the browser
Trade-off: Fast to launch, but can slow pages and flicker
Server-side
Pricing, checkout logic, search results, onboarding
Trade-off: Cleaner and faster for users, but needs engineering
Multivariate
Several elements and their combinations at once
Trade-off: Needs much more traffic and is harder to read
Multi-page
A change across a journey, such as category to cart
Trade-off: Shows the full effect, but tracking must cover every step
Reading the result
A dashboard declaring a winner is the start of the analysis, not the end. Work through three questions:
What happened?
Did the primary metric go up, down or stay flat, and what did the guardrails do?
How confident are we?
Did the test reach its planned sample size, and was anything unusual going on, such as a campaign, a stock problem or a tracking bug?
What do we do next?
Roll out the winner, iterate with a sharper hypothesis, or record why the idea failed.
Segments, such as mobile versus desktop, are useful for new hypotheses. Small slices swing a lot, so don't use them to rescue a losing test. After rollout, keep watching the metric, because effects sometimes fade once everyone sees the change.
A pre-launch checklist
Check each variant before it receives real traffic:
- Targeting: The right audience and exclusions are set.
- Tracking: Conversion events and revenue values fire correctly.
- Consistency: A visitor sees the same version after a refresh or return visit.
- Devices: Mobile and desktop layouts both work.
- Speed: The testing script does not slow the page noticeably.
- Conflicts: No clashes with pop-ups, personalization or other tests on the same page.
- Documentation: The hypothesis, primary metric and decision rule are written down.
Testing without harming users
A test that lifts clicks by confusing people is not an improvement. Ask of every variant:
- Clarity: Does it make the page easier to understand?
- Effort: Does it remove effort or doubt rather than add it?
- Honest expectations: Does the user know what happens after they click?
- Accessibility: Does it work with screen readers, keyboards and small screens?
- Nothing hidden: Are fees, terms and important details still visible?
- Long-term trust: Would a customer feel fairly treated if they saw both versions?
Common A/B testing mistakes
Most problems come from impatience or a fuzzy question:
- 1
No hypothesis. Random changes rarely teach anything that lasts.
- 2
Stopping early. Results swing a lot before enough data arrives.
- 3
Changing the test midway. A new audience, goal or design after launch spoils the comparison.
- 4
Ignoring guardrails. A lift in clicks hides a drop in revenue or satisfaction.
- 5
Overlapping tests. Two tests on the same page and audience interfere with each other.
- 6
Treating flat results as failures. A neutral test shows that the concern mattered less than expected.
- 7
Copying competitors. What worked for another audience may not work for yours.
- 8
Testing only to confirm opinions. If every result matches what the team already believed, the questions are too safe.
Building a testing habit
One test improves a page. A steady program improves how a team decides. Keep a backlog of ideas from analytics, support tickets, sales calls and research. Score each idea on impact, confidence and effort, and run the top ones in turn.
Record every test in the same format: hypothesis, audience, dates, primary metric, result, decision and next step. Over a year, that log becomes a record of what your customers respond to. Pair it with actionable metrics and cohort analysis so wins are judged on retention and revenue, not only on the first click.
Frequently asked questions
What is A/B testing?
How long should an A/B test run?
What is a guardrail metric?
What is the difference between A/B testing and multivariate testing?
Choosing A/B Testing Tools?
Read the buying guide: what to look for, mistakes to avoid and questions to ask vendors.
Related guides
- Cohort analysis: how to build and read a retention tableCohort analysis explained: cohort types, starting events, retention and churn metrics, how to read a retention table, a spreadsheet method and mistakes.Read the guide
- Actionable metrics vs vanity metrics, with examplesActionable metrics explained: how they differ from vanity metrics, the three A's, examples by team, flow metrics for predictability and a checklist.Read the guide
- Customer discovery: process, questions and decision treeCustomer discovery for startups: map assumptions, interview without leading, score needs, build evidence-based personas, read buying signals and decide what next.Read the guide
- Minimum viable product (MVP): meaning, types and examplesMVP meaning explained: what a minimum viable product is, the main MVP types, how to build one step by step, what to measure, and mistakes to avoid.Read the guide
Sources
Books and papers show their publication year. Web pages show the month we last checked them.
- 1.Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Cambridge University Press, 2020)
- 2.Ron Kohavi and Stefan Thomke, The Surprising Power of Online Experiments (Harvard Business Review, September-October 2017)
- 3.Eric Ries, The Lean Startup (Crown Business, 2011)
- 4.Jakob Nielsen, Why You Only Need to Test with 5 Users (Nielsen Norman Group, March 2000)