Statistical Significance Calculator

A pooled two-proportion z-test. The pooled rate p̂ = (x₁ + x₂) ÷ (n₁ + n₂), the standard error is √(p̂(1−p̂)(1/n₁ + 1/n₂)), and z is the difference in rates divided by that standard error.

The p-value is two-tailed: 2(1 − Φ(|z|)). One-tailed tests halve the p-value and flatter whichever variant happens to be ahead, so they're not offered here.

A p-value is the probability of a difference at least this large if the two variants were genuinely identical. It is not the probability that B beats A.

Required sample size per variant (the duration calculator) uses the standard two-proportion formula: n = (z₁₋α/₂ √(2p̄(1−p̄)) + z₁₋β √(p₁(1−p₁) + p₂(1−p₂)))² ÷ (p₂ − p₁)², with α = 0.05 (95% confidence) and β = 0.2 (80% power). Days = n × 2 ÷ daily visitors, rounded up to whole weeks so weekday and weekend behaviour are both represented.

This calculator runs a pooled two-proportion z-test, the standard large-sample test for comparing two conversion rates (NIST's engineering statistics handbook describes the same test for comparing two defect rates). It asks one question: if A and B truly converted at the same rate, how often would random assignment alone produce a gap as large as the one you saw?

The steps are: work out each variant's rate, pool both groups into a single rate that assumes no difference, use the pooled rate to find the standard error of the difference, then divide the observed difference by that standard error to get z. The p-value is the probability of a z at least that far from zero, in either direction.

Illustration, using the calculator's default numbers. Variant A: 2,000 visitors, 240 conversions, so 12.00%. Variant B: 2,000 visitors, 300 conversions, so 15.00%. Relative lift is (15 − 12) ÷ 12 = +25%.

Pooled rate: (240 + 300) ÷ (2,000 + 2,000) = 540 ÷ 4,000 = 13.5%. Standard error: √(0.135 × 0.865 × (1/2,000 + 1/2,000)) = √(0.116775 × 0.001) = 0.01081. z = (0.15 − 0.12) ÷ 0.01081 = 2.78. Two-tailed p = 2 × (1 − Φ(2.78)) ≈ 0.0055, well under 0.05, so the calculator reports Significant.

Now keep the same two rates but shrink the test to 200 visitors each (24 vs 30 conversions). The standard error grows to 0.0342, z falls to 0.88 and p rises to 0.38. Same 25% lift, no evidence at all. Sample size, not the size of the gap, is usually what decides the verdict.

One more: 10,000 visitors each, 1,200 vs 1,320 conversions (12.0% vs 13.2%, a 10% lift). z = 2.56, p ≈ 0.011. Significant, but only because the sample is large; a 10% lift is small and needs a lot of traffic to see.

A p-value of 0.03 means: if the variants were identical, a difference this large or larger would appear about 3 times in 100 tests. It does not mean there is a 97% chance B is better, and it says nothing about whether the lift is big enough to matter. The American Statistical Association's 2016 statement makes both points and warns against treating 0.05 as a bright line.

In practice, read three numbers together: the p-value (is there evidence of any difference?), the lift (how big is it?), and the sample (was the test sized to detect a lift this big?). A significant 0.3% lift on a pricing page and a non-significant 30% lift on 150 visitors are both results you should not ship on.

This calculator is two-tailed only. A one-tailed test counts only differences in the direction you predicted, which halves the p-value: z = 1.80 gives p = 0.072 two-tailed but 0.036 one-tailed, turning Not significant into Significant without a single extra visitor.

One-tailed tests are defensible when a result in the other direction would lead to exactly the same decision as no result, and when the direction is chosen before the data arrive. In product testing you almost always care if B is worse, so two-tailed is the honest default.

The p-value assumes you look once, at a sample size you fixed in advance. Checking every day and stopping at the first significant reading gives chance many extra opportunities to cross the line. In Evan Miller's worked example, a test at a nominal 5% level that is stopped as soon as it looks significant produces false positives 26.1% of the time, more than five times the rate you think you are running.

The fix is dull: decide the sample size first, then do not act on the result until you reach it. If you genuinely need to stop early, use a method built for it, such as a sequential design (Miller publishes a simple one) or the always-valid approaches covered in Kohavi, Tang and Xu's Trustworthy Online Controlled Experiments.

The second calculator on this page answers that before you start. It needs three inputs: your current conversion rate, the smallest relative lift worth detecting (the minimum detectable effect), and daily visitors across both variants. It fixes significance at 95% and power at 80%, meaning that if the true lift is exactly the size you entered, the test will catch it four times in five.

Illustration with its defaults: a 12% baseline and a 10% relative lift (12% to 13.2%) need 12,004 visitors per variant, 24,008 in total. At 500 visitors a day that is 48 days, which the calculator rounds up to 49 days, seven full weeks, so every weekday is represented equally.

Other calculators may use a different test (unpooled, one-tailed or Bayesian), so the same data can produce a slightly different p-value elsewhere. The order of magnitude is what matters: halving the lift you want to detect roughly quadruples the sample.

Every extra comparison is another chance for noise to look like a win. If you run k independent comparisons at the 0.05 level and none of the changes do anything, the chance of at least one false winner is 1 − 0.95^k: 14.3% for 3 comparisons, 22.6% for 5, and 40.1% for 10. Comparing three challengers to a control, or one test across ten metrics, falls into this trap.

Two standard corrections. Bonferroni divides the threshold by the number of comparisons (0.05 ÷ 5 = 0.01 each), which is simple and conservative. Benjamini and Hochberg's 1995 procedure controls the false discovery rate instead and keeps more power when you test many things. Either way, name one primary metric before the test starts.

For forms, the conversion you usually compare is submissions ÷ visitors who saw the form. Split traffic with your site's testing tool or a redirect that assigns visitors at random, and send each group to a different version of the form.

In Zunoform, the simplest setup is two copies of the form, each with its own link. Each form's Summary tab shows views, starts and completed submissions (the owner's own visits are not counted), which gives you the visitors and conversions to enter above. If both variants must feed one form, declare a hidden field such as variant and put ?variant=b in the B link; the value is stored with each response and exported with it. Zunoform does not randomize the split for you.

Once you know which stage is losing people, our form conversion rate calculator separates start rate from completion rate. Forms worth testing first are usually the high-traffic ones, like a lead capture form, a demo request or a newsletter signup.