Calculators
A/B Test Significance Calculator
Find out whether variant B really beat variant A, by how much, and how many visitors a reliable test needs.
- Runs in your browser
- Nothing uploaded
The tool
Enter visitors and conversions for each variant, or switch to sample size to plan a test before it starts.
Everything you enter stays on your own machine. Nothing is sent to us or to anyone else, which you can confirm in your browser network tab.
What people use it for
- Deciding whether a landing page or ad test has a real winner
- Planning how long a test must run before its result can be trusted
- Avoiding stopping a test early on a lucky streak
- Explaining a result to stakeholders with a confidence interval rather than a guess
- Checking the significance claims in a testing tool or agency report
About this tool
What statistical significance means in an A/B test
A result is statistically significant when the difference between two variants is larger than random chance would plausibly produce. This calculator uses a two-tailed, two-proportion z-test. It reports a p-value: the probability of seeing a difference at least this large if both variants actually performed the same. At 95% confidence, a p-value below 0.05 counts as significant.
The formula
Each variant has a conversion rate p = conversions ÷ visitors, and the pooled rate p̂ combines both variants. The test statistic is z = (pB − pA) ÷ √(p̂(1 − p̂)(1/nA + 1/nB)), and the p-value comes from the standard normal distribution. The confidence interval for the difference uses the unpooled standard error.
Worked example
Variant A converts 500 of 10,000 visitors (5.00%). Variant B converts 580 of 10,000 (5.80%), a relative uplift of 16%. The z-score is about 2.50 and the two-tailed p-value about 0.012, so the result is significant at 95% confidence but not at 99%.
Plan the sample size first
Most failed tests are underpowered rather than wrong. Before you start, enter your baseline conversion rate and the smallest uplift worth detecting. With a 5% baseline and a 20% relative uplift (5% to 6%), you need roughly 8,200 visitors per variant at 95% confidence and 80% power. Halve the uplift you want to detect and the required sample roughly quadruples.
Rules that keep tests honest
- Decide the sample size in advance, and do not stop the moment the result looks significant. Repeatedly checking and stopping early inflates false positives.
- Run tests for whole weeks so weekday and weekend behaviour are both included.
- Change one meaningful thing per test, and judge it on one primary metric.
- Treat a non-significant result as not proven, rather than as proof of no difference.
Better tests start with better traffic. Plan paid traffic with the ad performance calculator, measure it with the CPM, CPC and CTR calculator, and tag each variant link with the UTM link builder. If you want a conversion-focused redesign tested properly, our UI and UX design and web design and development teams can help.
More tools and services
- 1031 Exchange Calculator
- 401(k) Contribution Optimizer
- 529 College Savings Calculator
- Accessible Ramp Calculator
- Accounts Payable Aging Calculator
- Accounts Payable Days Calculator
If you would rather someone did this properly for you, we offer Web Design Services and Website Development, SEO Services: Search Engine Optimization, AI Automation Agency.
Questions
Questions about this tool
The things clients actually want to know — costs, timelines, ownership and what happens after launch.
Compare the p-value with your significance level. At 95% confidence the result is significant when the p-value is below 0.05, which this calculator flags automatically.
95% is the common default. Use 99% for expensive or hard-to-reverse changes, and 90% only for low-risk tests that are easy to undo.
It depends on your baseline conversion rate and the smallest uplift you care about. A 5% baseline and a 20% relative uplift needs about 8,200 visitors per variant at 95% confidence and 80% power.
No. Checking repeatedly and stopping at the first significant reading greatly increases false positives. Fix the sample size in advance and evaluate once it is reached.
It shows the range of differences consistent with your data. If the interval includes zero, the test has not shown a difference in either direction.
A two-tailed, two-proportion z-test with a pooled standard error, the standard approach for comparing conversion rates with reasonably large samples.
Free strategy session
Let’s build something worth finding.
Tell us where you want the business to be in twelve months. We will come back with a costed plan, a realistic timeline, and the numbers we expect to move — no obligation, no jargon.
- 30-minute call, no pitch deck
- Fixed-price proposal in 48 hours
- You own everything we build
Talk to a developer
Not a sales team
- Response time
- Within 1 business hour
- Where we work
- UK, USA & the Gulf
- Contracts
- No lock-in, monthly rolling
- Working since
- 2023