A/B Test Calculator

Q: What sample size do I need for an A/B test?

It depends on baseline rate and minimum detectable effect (MDE). For a 5% baseline conversion detecting a 10% relative lift at 95% confidence with 80% power, you need roughly 31,000 visitors per variant. Online calculators (or Evan Miller's) compute this from baseline rate, MDE, alpha, and power.

Q: When should I stop an A/B test?

Run tests for at least 1-2 full business cycles (typically 1-2 weeks) AND until you reach predetermined sample size. Avoid "peeking" - looking at results early inflates false positive rates. Use sequential testing methods (Bayesian, mSPRT) if you must monitor mid-test.

Q: What's a good minimum detectable effect (MDE)?

For mature products: 1-3% relative lift. For early-stage products with low traffic: 10-20%+. Smaller MDE requires larger sample sizes - detecting a 1% lift needs ~100x the traffic of detecting a 10% lift. Set MDE based on what would be commercially meaningful, not what's detectable.

Frequently Asked Questions

How is statistical significance calculated for A/B tests?

A/B tests typically use a two-proportion z-test. The formula computes z = (p1 - p2) / sqrt(p_pool × (1-p_pool) × (1/n1 + 1/n2)), where p1 and p2 are conversion rates and n1/n2 are sample sizes. A z-value above 1.96 means p < 0.05 (95% confidence).

What sample size do I need for an A/B test?

It depends on baseline rate and minimum detectable effect (MDE). For a 5% baseline conversion detecting a 10% relative lift at 95% confidence with 80% power, you need roughly 31,000 visitors per variant. Online calculators (or Evan Miller's) compute this from baseline rate, MDE, alpha, and power.

When should I stop an A/B test?

Run tests for at least 1-2 full business cycles (typically 1-2 weeks) AND until you reach predetermined sample size. Avoid "peeking" - looking at results early inflates false positive rates. Use sequential testing methods (Bayesian, mSPRT) if you must monitor mid-test.

What's a good minimum detectable effect (MDE)?

For mature products: 1-3% relative lift. For early-stage products with low traffic: 10-20%+. Smaller MDE requires larger sample sizes - detecting a 1% lift needs ~100x the traffic of detecting a 10% lift. Set MDE based on what would be commercially meaningful, not what's detectable.

Provided by AllCalculators.io
Free online calculators for everyday. No registration required.

Estimates only. Not professional business advice.

Business Information Disclaimer: Estimates only. Not professional business advice.

This calculator provides estimates for informational purposes only. Business results vary by industry, market conditions, and execution. Not a substitute for professional business consulting, accounting, or legal advice. Consult qualified professionals before making business decisions.

JavaScript is required to use the interactive calculator above. The questions and answers below remain readable without JavaScript.

How This Calculator Works

Enter the visitors and conversions for your control (A) and variant (B). The tool computes each conversion rate, then runs a two-proportion z-test: it pools the two rates, derives the standard error from p(1−p)(1/n₁ + 1/n₂), and converts the resulting z-score into a two-tailed p-value. Statistical confidence is reported as (1 − p) × 100, and the result is flagged "significant" only when that confidence meets or exceeds the threshold you select (90%, 95%, or 99%). It also reports the relative uplift, (rate₃ − rate₁) ÷ rate₁, and names the higher-converting variant.

Statistical Significance Is Not Business Significance

A "95% significant" result means roughly a 5% chance the observed gap is pure noise - it says nothing about whether the lift is large enough to matter. A 0.2% absolute improvement can be statistically significant with a huge sample yet commercially irrelevant. Always read the uplift and the confidence interval together: a tight interval around +0.1% is a real but trivial win; a wide interval spanning −5% to +30% means you simply don't have enough data yet.

The Peeking Problem

The single biggest source of false A/B wins is checking results repeatedly and stopping the moment p < 0.05. Each peek is another roll of the dice, so "continuous monitoring with early stopping" can inflate your real false-positive rate from 5% to 20–30%. Decide the sample size and test duration before launch and do not stop early unless you are using a sequential-testing method designed for it.

Sample Size and Duration Rules of Thumb

Detecting smaller lifts needs disproportionately more traffic - halving the minimum detectable effect roughly quadruples the required sample.

Run for full weekly cycles (ideally 1–4 weeks). Weekday vs. weekend and payday behavior differ enough to bias a 3-day test.

Very low base rates (e.g. 1% checkout) need large absolute volumes before the z-test approximation is trustworthy - aim for at least a few hundred conversions per arm.

Use 95% as the default; reserve 99% for high-stakes, hard-to-reverse changes.

Common Mistakes & Limitations

This is a fixed-horizon test for a single binary metric. Testing many metrics or variants at once multiplies false positives unless you correct for it.

The z-test assumes independent visitors; logged-in users on multiple devices or bot traffic violate that and skew results.

A non-significant result is not proof the variants are equal - it often just means underpowered.

Segment effects (mobile vs. desktop, new vs. returning) can hide inside a flat aggregate; a directionally "losing" test may win for a key segment.

A/B Test Calculator

Frequently Asked Questions

Related Business Calculators