← All Base Form Math projects
BM-25NOT STARTEDBASE FORM · PROBABILITYMATHPYTHON

A/B test calculator

Paste two conversion counts and get a clear answer: frequentist test, bootstrap interval and Bayesian chance-to-win, side by side.

EST.6 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
p-values, intervals, posteriors
GOAL · match within 1e-6
scipy.stats · conjugate formulas—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

An A/B test shows two versions of something to two random groups and counts how many people convert in each. B looks better. The real question is whether it is better, or just lucky this time. Your calculator answers that three ways and puts them side by side, so you can see where they agree and how they differ.

The frequentist answer is a hypothesis test. Assume no difference, pool the two groups to get one rate p^\hat p, and compute z=(p^B−p^A)/p^(1−p^)(1/nA+1/nB)z = (\hat p_B - \hat p_A) / \sqrt{\hat p (1 - \hat p)(1/n_A + 1/n_B)}. The p-value is the chance of a zz this far from zero if there really were no difference. The bootstrap gives a confidence interval for the difference without any formula: resample, recompute, repeat.

The Bayesian answer treats each true rate as uncertain. With a Beta(α\alpha, β\beta) prior and xx conversions out of nn, Bayes' rule gives a Beta(α+x\alpha + x, β+n−x\beta + n - x) posterior. From the two posteriors you can ask the question people actually want answered: what is the chance B's rate is higher than A's? That is P(pB>pA)=∫01fB(t) FA(t) dtP(p_B > p_A) = \int_0^1 f_B(t)\, F_A(t)\, dt, where fBf_B is B's posterior density and FAF_A is A's posterior CDF.

VISUALFig. 1
0.10 0.12 0.14 0.16 0.18 conversion rate A: 120 / 1,000 B: 145 / 1,000
Posteriors with a Beta(1, 1) prior after A converts 120 of 1,000 and B converts 145 of 1,000. The chance B's rate is higher is about 0.95. Why it's here: Seeing the overlap makes 'chance to win' concrete before you compute it.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

DATA · public A/B datasets

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your p-value matches scipy.stats.chi2_contingency (correction=False) and your chance-to-win by integration matches the closed-form conjugate sum, both within 1e-6. Record the larger gap with `python tools/mlarena.py record BM-25 "<max abs difference>"`.
  2. On 1,000 simulated A/A tests, about 5% come out significant at the 5% level.
  3. Run on a public A/B dataset, the report shows all three answers side by side with a one-sentence plain-English verdict.

Side quests

Simulate peeking. Check the p-value after every 100 visitors and stop at the first p below 0.05. Measure how much the false-positive rate grows.
Add a sample-size calculator. Given a base rate and the lift you care about, how many visitors per group give 80% power?
Add Fisher's exact test (scipy.stats.fisher_exact) for small counts and compare it with the z-test.

Terms in plain words

p-value
If there were truly no difference, the chance of seeing a gap at least as large as yours. It is not the chance that B is better.
Confidence interval
A range built so that, over many repeated experiments, 95% of such ranges contain the true difference.
Bootstrap
Resampling your own data with replacement many times to see how much a statistic would vary from sample to sample.
Prior and posterior
The prior is what you believe about a rate before the data. Bayes' rule combines it with the data to give the posterior.
Conjugate prior
A prior that gives a posterior of the same family. A Beta prior with yes/no data gives a Beta posterior, so the update is just adding counts.