Brief
Port PL-A09's AlphaZero to Hex with board sizes 3–9. Fit Elo-vs-compute frontiers (the best playing strength you can reach at each training-compute budget) and the train/test-compute trade-off (how much extra search at play time can stand in for extra training). Test Jones's claim that fits on small boards predict the frontiers on large boards. Then repeat on a second game family with a size knob (e.g., Connect-k on m×n boards) and publish whether the law transfers.
Approach
Elo turns match results into one number per agent. The chance that beats is , and ratings are fitted to make the observed results as likely as possible. Elo only fixes differences, so the paper pins MoHex, a perfect player on boards up to 9×9, at 0. Every other rating is then "Elo below perfect play".
The compute frontier for board size is the best Elo any snapshot reached at each training compute . Jones fits all board sizes at once with a five-parameter change-point model:
The paper reads its fit as about 500 Elo per decade of compute. Twice the compute is then worth Elo, about a 70% win rate, the "2/3 of the time" in the paper. The compute needed for perfect play grows about 7× per step in board size. The train-test trade-off on 9×9 is , so 10× more training saves about test-time search.
Budget first. Jones used about 500 GPU-hours on RTX 2080 Tis for training and 100 for evaluation, over roughly 200 models and 2,800 snapshots. The per-run compute caps in his Table II go from FLOPs on 3×3 to on 9×9, so boards 3 to 6 together cost under 1% of one maximal 9×9 run. Build the whole pipeline on those, check the frontier shape, and only then spend on 8 and 9. Evaluation is the hidden cost: every pair of snapshots playing 1,024 games is 700 million games at 2,800 snapshots. Batch games on the GPU as the paper does.
Where people get stuck
- Fitting runs instead of the frontier. The model describes the upper envelope, not individual runs. Bin compute on a log scale per board, take the maximum Elo in each bin, then fit. Fitting all points pulls the incline down.
- Elo that doesn't calibrate. Elo assumes strength is transitive. Self-play snapshots from one run can be non-transitive. Check the calibration plot before you trust any fit.
- Bad anchoring. If your best 9×9 agent never draws level with MoHex, everything is offset by an unknown constant. Play colour-balanced matches, since Hex without the swap rule favours the first player.
- Draws in the second game. Connect-k games can be drawn, and with perfect play many small boards are. Score a draw as half a win, and expect the ceiling to behave differently from Hex's.
- Too few snapshots near the knee. The plateau and the incline meet at a point that the fit has to locate. If your snapshot schedule is sparse there, the change point and the slope trade off against each other.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- The 9×9 frontier predicted from boards up to 7×7 lands within the error the paper reports in its Figure 7, recorded with `python tools/mlarena.py record PL-W02 "<residual variance at 9×9>"`.
- Fitted incline slope and perfect-play compute growth reported with bootstrap intervals, next to the paper's 500 Elo per decade and 7× per board size.
- The second-game result, positive or negative, published with code, match logs and fits.
Side quests
Terms in plain words
- Scaling law
- A simple formula that predicts performance from model size, data or compute, fitted on small runs and used to predict big ones.
- Compute frontier
- For each compute budget, the best result any run achieved; the curve along the top of all runs.
- Elo
- A rating where a fixed gap in points means a fixed chance of winning, so strength can be compared across agents.
- Train/test-compute trade-off
- How spending more search time per move during play can make up for less training, and vice versa.
- Hex
- A board game where players try to connect opposite sides; its board size is easy to change.
- Extrapolation error
- How far off a prediction is when you apply a fit outside the range of data it was fitted on.
- MoHex
- A Hex program that plays perfectly on boards up to 9×9; the paper fixes its rating at 0 Elo.
- Change-point model
- A curve made of straight pieces joined at points the fit chooses: here a flat plateau, a straight incline and a ceiling.
Why it rates ★★★★☆
Scaling-law methodology on a cheap testbed is what research leads care about, and the second game makes it new.