A/B Test Result Checker
Did my test really win, or was it luck?
Also known as: A/B test significance calculator
Test results
Significant at 95%
- Relative lift
- 22.5%
- 95% interval for the lift
- 2.1% to 42.9%
- Conversion rates
- 2% → 2.45%
- Chance B beats A
- 98.4%
Bayesian, flat prior.
Save this result
Keep it in your workspace with a note. Free accounts keep 5 results; Pro keeps them all.
How it works
How does the A/B Test Result Checker work?
A two-proportion z-test compares the two conversion rates; the p-value is the chance of a difference this large if the versions were really the same. The confidence interval shows the plausible range of the true lift.
- z = (p₂ − p₁) ÷ √(p̄(1−p̄)(1/n₁ + 1/n₂)) with the pooled rate p̄; p-value is two-sided.
- The confidence interval for the difference uses the unpooled standard error.
- Probability to beat control uses Beta(1+conversions, 1+non-conversions) posteriors (a flat prior), approximated as normal.
- Stopping before the planned sample size raises the false-positive rate, so the tool flags it.
Worked example
Control 200 conversions from 10,000 visitors (2.00%); variant 245 from 10,000 (2.45%). The lift is 22.5%, z = 2.16, p = 0.031: significant at 95%. The 95% interval for the lift is 2.1% to 42.9%, and the chance the variant is better is 98.4%.
Limits
Statistical significance isn't business significance: a real but tiny lift may not be worth shipping. The normal approximation needs reasonably large counts.
Related tools
Private by default
Your inputs stay in this browser.
This free calculation runs locally and is a planning aid, not financial, tax or legal advice. Nothing you type leaves this page unless you choose to save it.