Brief
Chatbot Arena ranks chat models from human votes on head-to-head battles. Its core is the Bradley-Terry model: each model has a strength score, and the chance that A beats B depends on the difference of their scores. Fit Bradley-Terry, and Rao-Kupper (an extension that allows ties), by MM (minorise-maximise, a simple iterative update that always increases the likelihood) and by Newton's method, on the public arena-human-preference-55k battles. Compute three kinds of intervals: sandwich (a standard-error formula that stays valid when the model is somewhat wrong), bootstrap, and Bayesian (Laplace approximation and MCMC). Add position bias (preferring the answer shown first) and length bias (preferring longer answers) as covariates. Validate interval coverage in a simulation with known strengths, then determine which adjacent pairs can be told apart.
Approach
Write each model's strength as . Bradley-Terry says
Only differences matter, so the model is unchanged if you add a constant to every . Fix that before anything else: anchor one model at 0 or force the strengths to sum to 0, and use the same anchor everywhere. Arena reports , which is only a rescaling. Hunter's MM update for a model with wins and games against is
followed by renormalisation. It always raises the likelihood but can be slow; Newton converges in a handful of steps once it is close. Rao-Kupper adds a tie parameter : and .
For intervals, let be the Hessian of the negative log-likelihood and the gradient contribution of battle . The usual standard errors come from ; the sandwich uses , which stays valid when the model is somewhat wrong. Covariates go inside the logistic: , where is the first-position advantage. Build the fit and its reference check first, then intervals, then the coverage simulation, and only then look at which real pairs separate.
Where people get stuck
- No anchor. Without a constraint the Hessian is singular, Newton fails, and standard errors come out infinite or arbitrary. Every interval depends on the anchor you chose, so compare differences, which do not.
- Overlap as a test. Two 95% intervals can overlap while the difference is still significant. Use .
- Too few simulations. Coverage from datasets has a standard error of . With 200 runs that is 1.5%, too coarse to check 95 ± 1%. Plan for 2,000 or more.
- A different design in simulation. Real battles are unbalanced: popular models play far more. Simulate on the real pairing design, or your coverage says nothing about the real intervals.
- Raw text columns. The prompt and response columns store the conversation as an encoded list of turns. Decode them before you measure length, and decide whether length means characters or tokens.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- MLE matches a scipy-optimised logistic reference to 1e-8; intervals reach 95 ± 1% coverage in simulation; the fraction of separable adjacent pairs is published with and without length-bias control, recorded with `python tools/mlarena.py record PM-A02 "<k>/<n> pairs separable"`.
- Coverage is estimated from at least 2,000 simulated datasets, so its own Monte Carlo error is about 0.5%.
- The position and length coefficients are reported with intervals, and you say in one sentence how much the length control changes the ranking.
Side quests
Terms in plain words
- Bradley-Terry model
- A model for pairwise contests where the probability that A beats B is a logistic function of the difference in their strengths.
- Rao-Kupper model
- An extension of Bradley-Terry with an extra parameter that gives ties their own probability.
- MM algorithm
- An optimisation method that repeatedly maximises a simple lower bound of the likelihood, so every step improves the fit.
- Sandwich estimator
- A robust formula for standard errors that does not rely on the model being exactly correct.
- Laplace approximation
- Approximating a posterior by a Gaussian centred at its peak, with spread taken from the curvature there.
- Covariate
- An extra input variable, such as answer length, included in the model so its effect is separated from model strength.
- Coverage
- The share of simulated datasets in which a 95% interval actually contains the true value. A correct method gives about 95%.
Why it rates ★★★☆☆
ML-relevant statistics done right, though re-analysing leaderboards is well-trodden.