← All Cracked ML projects
PL-W02NOT STARTEDWOWMLPYTHONIMPACT ★★★★☆

Scaling scaling laws: Hex, then a game nobody fitted

Predict how much compute a 9×9 board needs using only small boards, then test whether the law holds in a new game.

EST.60 h
REWARD+600 XP
AFTERAndy L. Jones + 1

ML-ARENA BENCHMARKS

PROOF: FINDING
METRICINDUSTRYMINE
extrapolation error of the 9×9 compute frontier when it is predicted from fits on boards ≤ 7×7; fit quality on the new game
GOAL · reproduce the frontier shape and predict 9×9 within the paper's reported error band; publish the second-game result (positive or negative)
Jones 2021, Scaling Scaling Laws with Board Games (frontiers are exponential in performance and board size)—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Port PL-A09's AlphaZero to Hex with board sizes 3–9. Fit Elo-vs-compute frontiers (the best playing strength you can reach at each training-compute budget) and the train/test-compute trade-off (how much extra search at play time can stand in for extra training). Test Jones's claim that fits on small boards predict the frontiers on large boards. Then repeat on a second game family with a size knob (e.g., Connect-k on m×n boards) and publish whether the law transfers.

Approach

Elo turns match results into one number per agent. The chance that AA beats BB is P=1/(1+10−(EA−EB)/400)P = 1/(1 + 10^{-(E_A - E_B)/400}), and ratings are fitted to make the observed results as likely as possible. Elo only fixes differences, so the paper pins MoHex, a perfect player on boards up to 9×9, at 0. Every other rating is then "Elo below perfect play".

The compute frontier for board size bb is the best Elo any snapshot reached at each training compute CC. Jones fits all board sizes at once with a five-parameter change-point model:

plateau=mbp b+cp,incline=mbi b+mClog⁡10C+ci,Elo=min⁡(0, max⁡(plateau, incline)).\text{plateau} = m^{p}_{b}\, b + c^{p}, \qquad \text{incline} = m^{i}_{b}\, b + m_{C} \log_{10} C + c^{i}, \qquad \text{Elo} = \min\bigl(0,\ \max(\text{plateau},\ \text{incline})\bigr).

The paper reads its fit as about 500 Elo per decade of compute. Twice the compute is then worth 500log⁡102≈150500 \log_{10} 2 \approx 150 Elo, about a 70% win rate, the "2/3 of the time" in the paper. The compute needed for perfect play grows about 7× per step in board size. The train-test trade-off on 9×9 is log⁡10Ctest=−1.2log⁡10Ctrain+0.004 Elo+29\log_{10} C_{\text{test}} = -1.2 \log_{10} C_{\text{train}} + 0.004\,\text{Elo} + 29, so 10× more training saves about 101.2≈15×10^{1.2} \approx 15\times test-time search.

Budget first. Jones used about 500 GPU-hours on RTX 2080 Tis for training and 100 for evaluation, over roughly 200 models and 2,800 snapshots. The per-run compute caps in his Table II go from 101210^{12} FLOPs on 3×3 to 101710^{17} on 9×9, so boards 3 to 6 together cost under 1% of one maximal 9×9 run. Build the whole pipeline on those, check the frontier shape, and only then spend on 8 and 9. Evaluation is the hidden cost: every pair of snapshots playing 1,024 games is 700 million games at 2,800 snapshots. Batch games on the GPU as the paper does.

VISUALFig. 1
3×3 4×4 5×5 6×6 7×7 8×8 9×9 0 = perfect play incline: slope m_flops plateaus: random play, lower on bigger boards training compute (log scale) → Elo vs perfect play fittedboards 3–7 predictedboards 8–9, thencheck vs measured schematic: shapeonly, no values
Schematic of the change-point model. Each board size has a plateau at random play, a straight incline in log compute, and a ceiling at 0, perfect play. You fit the solid curves and predict the dashed ones. Why it's here: Objectives 5 and 6 are about fitting these three pieces and checking the dashed predictions, so see the shape before fitting it.

Where people get stuck

  • Fitting runs instead of the frontier. The model describes the upper envelope, not individual runs. Bin compute on a log scale per board, take the maximum Elo in each bin, then fit. Fitting all points pulls the incline down.
  • Elo that doesn't calibrate. Elo assumes strength is transitive. Self-play snapshots from one run can be non-transitive. Check the calibration plot before you trust any fit.
  • Bad anchoring. If your best 9×9 agent never draws level with MoHex, everything is offset by an unknown constant. Play colour-balanced matches, since Hex without the swap rule favours the first player.
  • Draws in the second game. Connect-k games can be drawn, and with perfect play many small boards are. Score a draw as half a win, and expect the ceiling to behave differently from Hex's.
  • Too few snapshots near the knee. The plateau and the incline meet at a point that the fit has to locate. If your snapshot schedule is sparse there, the change point and the slope trade off against each other.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 7
Stuck? Hints stay hidden until you ask.

Objectives

0 / 8 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. The 9×9 frontier predicted from boards up to 7×7 lands within the error the paper reports in its Figure 7, recorded with `python tools/mlarena.py record PL-W02 "<residual variance at 9×9>"`.
  2. Fitted incline slope and perfect-play compute growth reported with bootstrap intervals, next to the paper's 500 Elo per decade and 7× per board size.
  3. The second-game result, positive or negative, published with code, match logs and fits.

Side quests

Repeat the train-test trade-off on 9×9: evaluate snapshots with tree sizes from 1 to 512 nodes and fit log test-compute against log train-compute and Elo. Does your exchange rate match the paper's 1.2?
Test the paper's toy model, where twice the compute wins about 2/3 of games, on your second game. A game with draws may break it in an interesting way.

Terms in plain words

Scaling law
A simple formula that predicts performance from model size, data or compute, fitted on small runs and used to predict big ones.
Compute frontier
For each compute budget, the best result any run achieved; the curve along the top of all runs.
Elo
A rating where a fixed gap in points means a fixed chance of winning, so strength can be compared across agents.
Train/test-compute trade-off
How spending more search time per move during play can make up for less training, and vice versa.
Hex
A board game where players try to connect opposite sides; its board size is easy to change.
Extrapolation error
How far off a prediction is when you apply a fit outside the range of data it was fitted on.
MoHex
A Hex program that plays perfectly on boards up to 9×9; the paper fixes its rating at 0 Elo.
Change-point model
A curve made of straight pieces joined at points the fit chooses: here a flat plateau, a straight incline and a ceiling.

Why it rates ★★★★☆

Scaling-law methodology on a cheap testbed is what research leads care about, and the second game makes it new.