← All Cracked Math projects
PM-B07NOT STARTEDBEGINNERMATHPYTHONIMPACT ★★★☆☆

Error bars for evals: when is 'model A beats model B' real?

Most leaderboard gaps are noise. Build the tool that tells you which ones are.

EST.6 h
REWARD+200 XP
AFTEREvan Miller 2024

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
Empirical coverage of nominal 95% intervals across 40 simulated settings; minimum detectable difference (the smallest accuracy gap the test can reliably tell apart from noise) for a real model pair
GOAL · The recommended interval has 94-96% coverage in >= 38/40 settings. Wald is shown to fall below 90% coverage when n*p < 10. The paired test reports the minimum detectable difference.
Miller 2024, Adding Error Bars to Evals (arXiv 2411.00640) · coverage measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

A model's benchmark accuracy is an estimate from a finite set of questions, so it needs an error bar. Implement five kinds of confidence interval for benchmark accuracy: Wald (the textbook estimate plus or minus 1.96 standard errors), Wilson (a version that behaves better near 0% and 100%), clustered standard errors (for questions that come in related groups), paired differences (comparing two models on the same questions) and the bootstrap (resampling the questions many times). Run a coverage simulation to show where each interval breaks: 10^4 synthetic evals per setting, n = 50..10k questions, accuracy near 0 and 1, and clustered questions. Coverage is how often a nominal 95% interval really contains the true value. Then apply the best interval to the per-question outputs of two real published models.

VICTORY CONDITIONS

  1. The recommended interval has 94-96% coverage in >= 38/40 settings. Wald is shown to fall below 90% coverage when n*p < 10. The paired test reports the minimum detectable difference. (Empirical coverage of nominal 95% intervals across 40 simulated settings; minimum detectable difference (the smallest accuracy gap the test can reliably tell apart from noise) for a real model pair).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Confidence interval
A range computed from data that is designed to contain the true value a stated share of the time, such as 95%.
Coverage
How often, in repeated experiments, an interval actually contains the true value; a good 95% interval covers about 95% of the time.
Clustered standard error
An error bar that accounts for questions coming in correlated groups, which makes the effective sample size smaller than the question count.
Bootstrap
Estimating uncertainty by resampling your data with replacement many times and looking at how much the result varies.
Paired test
Comparing two models question by question on the same set, which cancels out shared question difficulty and gives tighter error bars.

Why it rates ★★★☆☆

Directly useful on any eval team. Most ML candidates get paired and clustered errors wrong.