Brief
A model's benchmark accuracy is an estimate from a finite set of questions, so it needs an error bar. Implement five kinds of confidence interval for benchmark accuracy: Wald (the textbook estimate plus or minus 1.96 standard errors), Wilson (a version that behaves better near 0% and 100%), clustered standard errors (for questions that come in related groups), paired differences (comparing two models on the same questions) and the bootstrap (resampling the questions many times). Run a coverage simulation to show where each interval breaks: 10^4 synthetic evals per setting, n = 50..10k questions, accuracy near 0 and 1, and clustered questions. Coverage is how often a nominal 95% interval really contains the true value. Then apply the best interval to the per-question outputs of two real published models.
VICTORY CONDITIONS
- The recommended interval has 94-96% coverage in >= 38/40 settings. Wald is shown to fall below 90% coverage when n*p < 10. The paired test reports the minimum detectable difference. (Empirical coverage of nominal 95% intervals across 40 simulated settings; minimum detectable difference (the smallest accuracy gap the test can reliably tell apart from noise) for a real model pair).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Confidence interval
- A range computed from data that is designed to contain the true value a stated share of the time, such as 95%.
- Coverage
- How often, in repeated experiments, an interval actually contains the true value; a good 95% interval covers about 95% of the time.
- Clustered standard error
- An error bar that accounts for questions coming in correlated groups, which makes the effective sample size smaller than the question count.
- Bootstrap
- Estimating uncertainty by resampling your data with replacement many times and looking at how much the result varies.
- Paired test
- Comparing two models question by question on the same set, which cancels out shared question difficulty and gives tighter error bars.
Why it rates ★★★☆☆
Directly useful on any eval team. Most ML candidates get paired and clustered errors wrong.