← All Cracked Math projects
PM-A02NOT STARTEDADVANCEDMATHPYTHONIMPACT ★★★☆☆

Which Chatbot Arena gaps are real? Bradley-Terry with honest uncertainty

Fit the Chatbot Arena leaderboard yourself and find out how many neighbouring ranks are statistically indistinguishable.

EST.22 h
REWARD+400 XP
AFTERChiang et al. 2024 + 2

ML-ARENA BENCHMARKS

PROOF: FINDING
METRICINDUSTRYMINE
MLE agreement with a reference solver; interval coverage in simulation; fraction of adjacent pairs separable at 95%
GOAL · MLE matches a scipy-optimised logistic reference to 1e-8; intervals reach 95 +/- 1% coverage in simulation; the fraction of separable adjacent pairs is published, with and without length-bias control
Chatbot Arena paper methodology (arXiv 2403.04132) · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Chatbot Arena ranks chat models from human votes on head-to-head battles. Its core is the Bradley-Terry model: each model has a strength score, and the chance that A beats B depends on the difference of their scores. Fit Bradley-Terry, and Rao-Kupper (an extension that allows ties), by MM (minorise-maximise, a simple iterative update that always increases the likelihood) and by Newton's method, on the public arena-human-preference-55k battles. Compute three kinds of intervals: sandwich (a standard-error formula that stays valid when the model is somewhat wrong), bootstrap, and Bayesian (Laplace approximation and MCMC). Add position bias (preferring the answer shown first) and length bias (preferring longer answers) as covariates. Validate interval coverage in a simulation with known strengths, then determine which adjacent pairs can be told apart.

Approach

Write each model's strength as θi\theta_i. Bradley-Terry says

P(i beats j)=11+e−(θi−θj)=γiγi+γj,γi=eθi.P(i \text{ beats } j) = \frac{1}{1 + e^{-(\theta_i - \theta_j)}} = \frac{\gamma_i}{\gamma_i + \gamma_j}, \qquad \gamma_i = e^{\theta_i}.

Only differences matter, so the model is unchanged if you add a constant to every θ\theta. Fix that before anything else: anchor one model at 0 or force the strengths to sum to 0, and use the same anchor everywhere. Arena reports 1000+400ln⁡10 θi1000 + \tfrac{400}{\ln 10}\,\theta_i, which is only a rescaling. Hunter's MM update for a model with WiW_i wins and NijN_{ij} games against jj is

γi←Wi∑j≠iNij/(γi+γj),\gamma_i \leftarrow \frac{W_i}{\sum_{j \ne i} N_{ij} / (\gamma_i + \gamma_j)} ,

followed by renormalisation. It always raises the likelihood but can be slow; Newton converges in a handful of steps once it is close. Rao-Kupper adds a tie parameter ν≥1\nu \ge 1: P(i≻j)=γi/(γi+νγj)P(i \succ j) = \gamma_i / (\gamma_i + \nu\gamma_j) and P(tie)=(ν2−1)γiγj/((γi+νγj)(γj+νγi))P(\text{tie}) = (\nu^2 - 1)\gamma_i\gamma_j / \big((\gamma_i + \nu\gamma_j)(\gamma_j + \nu\gamma_i)\big).

For intervals, let HH be the Hessian of the negative log-likelihood and gkg_k the gradient contribution of battle kk. The usual standard errors come from H−1H^{-1}; the sandwich uses H−1(∑kgkgk⊤)H−1H^{-1}\big(\sum_k g_k g_k^\top\big)H^{-1}, which stays valid when the model is somewhat wrong. Covariates go inside the logistic: P(a wins)=σ(θa−θb+h+β Δlen)P(a \text{ wins}) = \sigma(\theta_a - \theta_b + h + \beta\,\Delta_{\text{len}}), where hh is the first-position advantage. Build the fit and its reference check first, then intervals, then the coverage simulation, and only then look at which real pairs separate.

VISUALFig. 1
750 850 950 1050 1150 1250 Arena-style score (400/ln 10 × strength + 1000) rankvs next #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 separable, no overlap overlap, still separable not separable true
A simulated leaderboard: 12 models with known strengths, 5,485 random battles, 95% intervals from the Hessian. 6 of 11 neighbouring pairs are separable, and 2 of those have overlapping intervals. Why it's here: Overlapping intervals are not the test. The test is on the difference, which uses the covariance between the two estimates.

Where people get stuck

  • No anchor. Without a constraint the Hessian is singular, Newton fails, and standard errors come out infinite or arbitrary. Every interval depends on the anchor you chose, so compare differences, which do not.
  • Overlap as a test. Two 95% intervals can overlap while the difference is still significant. Use Var⁡(θ^a−θ^b)=Vaa+Vbb−2Vab\operatorname{Var}(\hat\theta_a - \hat\theta_b) = V_{aa} + V_{bb} - 2V_{ab}.
  • Too few simulations. Coverage from RR datasets has a standard error of 0.95×0.05/R\sqrt{0.95 \times 0.05 / R}. With 200 runs that is 1.5%, too coarse to check 95 ± 1%. Plan for 2,000 or more.
  • A different design in simulation. Real battles are unbalanced: popular models play far more. Simulate on the real pairing design, or your coverage says nothing about the real intervals.
  • Raw text columns. The prompt and response columns store the conversation as an encoded list of turns. Decode them before you measure length, and decide whether length means characters or tokens.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

Objectives

0 / 8 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. MLE matches a scipy-optimised logistic reference to 1e-8; intervals reach 95 ± 1% coverage in simulation; the fraction of separable adjacent pairs is published with and without length-bias control, recorded with `python tools/mlarena.py record PM-A02 "<k>/<n> pairs separable"`.
  2. Coverage is estimated from at least 2,000 simulated datasets, so its own Monte Carlo error is about 0.5%.
  3. The position and length coefficients are reported with intervals, and you say in one sentence how much the length control changes the ranking.

Side quests

Active sampling: for one inseparable adjacent pair, work out how many extra battles (and against which opponents) would most likely separate it, then check by simulation.
Put a hierarchical prior on the strengths and see whether shrinkage changes which pairs are separable.

Terms in plain words

Bradley-Terry model
A model for pairwise contests where the probability that A beats B is a logistic function of the difference in their strengths.
Rao-Kupper model
An extension of Bradley-Terry with an extra parameter that gives ties their own probability.
MM algorithm
An optimisation method that repeatedly maximises a simple lower bound of the likelihood, so every step improves the fit.
Sandwich estimator
A robust formula for standard errors that does not rely on the model being exactly correct.
Laplace approximation
Approximating a posterior by a Gaussian centred at its peak, with spread taken from the curvature there.
Covariate
An extra input variable, such as answer length, included in the model so its effect is separated from model strength.
Coverage
The share of simulated datasets in which a 95% interval actually contains the true value. A correct method gives about 95%.

Why it rates ★★★☆☆

ML-relevant statistics done right, though re-analysing leaderboards is well-trodden.