A/B Testing Significance Calculator

Two-proportion z-test with p-value, confidence intervals and lift. Everything computed in your browser — no data uploaded.

—

—

—

Statistical Breakdown

Values

How Significance Testing Works

Two proportions (A and B) rarely differ by exactly 0%. The question is whether the difference is real or noise. A two-proportion z-test computes the probability that a difference this large would appear if there were no real difference.

The Math

p̂A = cA / nA
p̂B = cB / nB
p̂pooled = (cA + cB) / (nA + nB)
SE = √(p̂pooled × (1 − p̂pooled) × (1/nA + 1/nB))
z = (p̂B − p̂A) / SE
p-value = 2 × (1 − Φ(|z|))

Reading the Result

  • p < 0.01 — Very strong evidence. Ship B.
  • p < 0.05 — Statistically significant (standard threshold).
  • p < 0.10 — Suggestive. Continue testing.
  • p ≥ 0.10 — No significant difference. Keep running.

Common Traps

  • Peeking early. Checking significance daily inflates false positives. Pre-commit to a sample size.
  • Multiple variants. Testing 5 variants? Adjust the threshold (Bonferroni) or you'll ship noise.
  • Small samples. Under ~1,000 visitors per variant, results are unstable.
  • Novelty effect. New designs often spike then settle. Wait 2+ weeks.

A/B Testing FAQ

What p-value means significance?

p < 0.05 is the standard threshold — 5% chance the difference is noise. For higher confidence, use p < 0.01.

Can I stop the test when it becomes significant?

No. Early peeking inflates false positives by 2–3x. Set a sample size upfront and wait until it's reached.

What's a minimum sample size?

Rough rule: 1,000+ visitors per variant for conversion rates around 5%. Higher base rates need fewer; lower base rates need more.

Does a significant result guarantee success?

No. It guarantees a real effect on this sample. Real-world performance can still differ due to novelty effects, seasonality or audience drift.