Brief
An A/B test shows two versions of something to two random groups and counts how many people convert in each. B looks better. The real question is whether it is better, or just lucky this time. Your calculator answers that three ways and puts them side by side, so you can see where they agree and how they differ.
The frequentist answer is a hypothesis test. Assume no difference, pool the two groups to get one rate , and compute . The p-value is the chance of a this far from zero if there really were no difference. The bootstrap gives a confidence interval for the difference without any formula: resample, recompute, repeat.
The Bayesian answer treats each true rate as uncertain. With a Beta(, ) prior and conversions out of , Bayes' rule gives a Beta(, ) posterior. From the two posteriors you can ask the question people actually want answered: what is the chance B's rate is higher than A's? That is , where is B's posterior density and is A's posterior CDF.
DATA · public A/B datasets
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your p-value matches scipy.stats.chi2_contingency (correction=False) and your chance-to-win by integration matches the closed-form conjugate sum, both within 1e-6. Record the larger gap with `python tools/mlarena.py record BM-25 "<max abs difference>"`.
- On 1,000 simulated A/A tests, about 5% come out significant at the 5% level.
- Run on a public A/B dataset, the report shows all three answers side by side with a one-sentence plain-English verdict.
Side quests
Terms in plain words
- p-value
- If there were truly no difference, the chance of seeing a gap at least as large as yours. It is not the chance that B is better.
- Confidence interval
- A range built so that, over many repeated experiments, 95% of such ranges contain the true difference.
- Bootstrap
- Resampling your own data with replacement many times to see how much a statistic would vary from sample to sample.
- Prior and posterior
- The prior is what you believe about a rate before the data. Bayes' rule combines it with the data to give the posterior.
- Conjugate prior
- A prior that gives a posterior of the same family. A Beta prior with yes/no data gives a Beta posterior, so the update is just adding counts.