Statistical Significance Calculator
Enter visitors and conversions for both variants of your A/B test. You'll get the p-value, the relative lift, and a straight answer on whether the difference clears the usual 95% bar. Below it, a second calculator tells you how many visitors you need before a result can mean anything — the number most people wish they had checked first.
Free, no signup, and everything is calculated in your browser.
p = 0.0055 · 99.4% confidence
Two-tailed pooled z-test. A p-value is the chance of seeing a difference this large if the two variants were actually identical — not the chance that B is better.
How long to run it
Sample size and test duration for the lift you hope to detect, at 95% confidence and 80% power.
24,008 visitors in total · 12,004 per variant · rounded up to whole weeks
How significance is calculated
A pooled two-proportion z-test. The pooled rate p̂ = (x₁ + x₂) ÷ (n₁ + n₂), the standard error is √(p̂(1−p̂)(1/n₁ + 1/n₂)), and z is the difference in rates divided by that standard error.
The p-value is two-tailed: 2(1 − Φ(|z|)). One-tailed tests halve the p-value and flatter whichever variant happens to be ahead, so they're not offered here.
A p-value is the probability of a difference at least this large if the two variants were genuinely identical. It is not the probability that B beats A.
Required sample size per variant (the duration calculator) uses the standard two-proportion formula: n = (z₁₋α/₂ √(2p̄(1−p̄)) + z₁₋β √(p₁(1−p₁) + p₂(1−p₂)))² ÷ (p₂ − p₁)², with α = 0.05 (95% confidence) and β = 0.2 (80% power). Days = n × 2 ÷ daily visitors, rounded up to whole weeks so weekday and weekend behaviour are both represented.
Questions people ask
How long should I run a test?
Decide the sample size before you start and don't stop early because a result looks good — checking repeatedly and stopping at the first significant reading is the most common way to produce a false positive.
My result isn't significant. Does that mean the variants are the same?
No. It means you don't have enough evidence to distinguish them. Absence of significance is not evidence of equivalence, especially at small samples.
What is a p-value, in plain terms?
The p-value is how surprising your result would be if the two variants were actually identical. A p-value of 0.03 means a gap this large would show up by chance about 3 times in 100 — rare enough that most teams treat it as real. The conventional cut-off is 0.05, which is where "95% confidence" comes from.
Is 90% confidence good enough?
It depends on the cost of being wrong. At 90% you will call a false winner about one test in ten. That is fine for a button colour and reckless for a pricing page. Decide the threshold before the test, not after you have seen which variant it favours.
How long should I run an A/B test?
Long enough to reach the sample size the duration calculator gives you, and never shorter than one full week — behaviour on Tuesday and Saturday differ, and a test that only saw one of them is measuring the day, not the variant. Stopping the moment a result turns significant is the most common way to ship a change that does nothing.
One-tailed or two-tailed?
Two-tailed, which is what this calculator uses. A one-tailed test assumes you already know which direction the difference goes; it halves the p-value and flatters whichever variant happens to be ahead. If you are honest about not knowing the outcome, two-tailed is the only defensible choice.
Does this work for email, surveys and ads, not just web pages?
Yes. The maths only cares about two groups, each with a count of people and a count of successes — opens against sends, completions against starts, clicks against impressions. Anything that can be expressed that way can be tested here.
Further reading
Related tools
Better forms.
Better data.
Build your first form in under 60 seconds. Free forever for personal use, no credit card required.