A/B Test Sample Size Calculator

Frequently Asked Questions

What is the minimum detectable effect?

The MDE is the smallest difference in conversion rates that your test is designed to detect with the specified power. It is expressed in absolute percentage points: an MDE of 2 pp means you want to detect a move from, say, 10% to 12%.

Should I use one-tailed or two-tailed significance?

Use a two-tailed test unless you have a strong, pre-registered reason to care only about one direction. Two-tailed tests are the default because a new variant can hurt conversion just as easily as it can help.

What happens if I stop the test early?

Stopping as soon as significance is reached inflates the actual false-positive rate above α. With α = 5% and a 50% early-stop rule, the true false-positive rate can exceed 25%. Always commit to the planned sample before launching.

Why does the calculator return per-variant and total sample?

Standard A/B tests split traffic 50/50, so each variant receives n visitors and the experiment requires 2n in total. If you run more than two variants, multiply the per-variant n by the number of arms.

Provided by AllCalculators.io
Free online calculators for everyday. No registration required.

Estimates for informational purposes only.

Important Disclaimer: Estimates for informational purposes only.

This calculator provides estimates for informational purposes only. Results are based on assumptions and may not reflect actual outcomes. Consult qualified professionals in relevant fields before making important decisions based on these results.

JavaScript is required to use the interactive calculator above. The questions and answers below remain readable without JavaScript.

How This Calculator Works

Before running an A/B test you need to know how many visitors each variant requires to detect a real difference rather than random noise. This tool uses the standard two-proportion sample-size formula based on four inputs: your current baseline conversion rate, the smallest lift you care about detecting (the minimum detectable effect, or MDE), your significance level α, and your desired statistical power 1−β.

The formula combines the critical z-scores for both error types: n = (z_α/2 + z_β)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ (p₂−p₁)². Here p₁ is the baseline rate, p₂ = p₁ + MDE is the rate under the alternative, and z-values are drawn from the inverse standard normal. The result is rounded up to the next whole visitor.

Significance Level and Statistical Power

The significance level α (commonly 5%) is the maximum tolerable false-positive rate: the probability of declaring a winner when the two variants are actually identical. Lowering α reduces false alarms but demands a larger sample.

The power (commonly 80%) is the probability of detecting a true difference of at least the MDE. Higher power is desirable but again requires more visitors. Power = 1−β, where β is the false-negative rate. A test run with only 50% power misses real effects half the time, making it nearly worthless.

Why the Minimum Detectable Effect Drives Everything

Sample size is inversely proportional to the square of the effect size. Halving the MDE you want to detect roughly quadruples the required sample. This is why picking a realistic, business-relevant minimum effect is the most important decision in A/B test planning.

If your baseline conversion is 10% and you only care about detecting lifts larger than 2 percentage points (p₂ = 12%), your MDE in absolute terms is 2 pp. Chasing a 0.5 pp effect with the same settings demands 16 times as many visitors - an impractical commitment for most sites.

Running a Valid Test

Run for full weeks. Traffic patterns vary by day of week. Stopping on a Saturday after launching on a Monday under-represents weekday behavior.

Do not stop early. Peeking at results and stopping as soon as significance flickers is called optional stopping, and it inflates the true false-positive rate far above α. Commit to the planned sample size before launch.

One metric at a time. Testing five metrics simultaneously and reporting only the significant one is the same as running five tests at α = 5% each - the combined false-positive rate is much higher.

Segment after, not before. Sub-group analysis after a non-significant result is exploratory only, not confirmatory.

Common Mistakes

The most frequent error is choosing an MDE that is too small because the researcher hopes to detect a tiny improvement, leading to sample sizes that take months to achieve. A close second is testing without a pre-registered hypothesis - declaring a winner on any metric that happened to move is post-hoc significance hunting.

Other mistakes include forgetting to account for novelty effects (users react to newness, not the change itself), ignoring traffic seasonality, or splitting traffic unequally when the formula assumes a 50/50 split. This calculator assumes equal allocation; for unequal splits, adjust the formula accordingly.

Interpreting Results

Once you have reached the planned sample size, a p-value below α means you can reject the null hypothesis that the conversion rates are equal - the observed difference is unlikely under chance alone. But significance is not the same as importance: always report the observed absolute and relative lift alongside the p-value and a confidence interval for the difference.

If the test is non-significant, that does not prove the variants are equal - it simply means you lacked enough evidence to conclude otherwise at your chosen α and power. A non-inferiority test or Bayesian analysis may be more appropriate when your goal is to confirm equivalence.

A/B Test Sample Size Calculator

Frequently Asked Questions

Related Statistics Calculators