Brief
Study the record history, profile the current record on 8×H100 (rented), and find a statistically significant improvement: an optimiser change (Muon variants), an architecture tweak, kernel fusion or a schedule change. Track 1 (GPT-2 124M → val loss 3.28) is now about 40 s, so the realistic targets are Track 2 (GPT-2 Medium → 2.92) or Track 3 (fewest steps). Rules: no data-pipeline changes, and p < 0.01 over multiple runs (the gain must be very unlikely to be luck).
Approach
Pick the track before anything else, because each has its own economics. Track 1 (GPT-2 Small to 3.28) stood at 0.665 minutes after record #92 on 2026-08-30. Its recent records come from FP8 kernels, sampled softmax, a hashed n-gram table and full CUDA-graph capture, so the remaining gains there are mostly systems work. Track 2 (GPT-2 Medium to 2.92) stood at 17.35 minutes after record #18 on 2025-12-31, with far fewer records so far. Track 3 fixes the architecture, data and batch size and counts steps to 3.28. Wall-clock time doesn't matter there, so it is an optimiser contest. Its best result was 2,690 steps (#46, 2026-06-19).
The statistics are part of the submission. For Tracks 1 and 2 you need a one-sided test that the mean loss over runs is below the target:
Track 3 uses a z-test with a fixed , which reduces to . The figure shows what that costs. With the run-to-run spread of 0.0019 measured over 80 runs in the January 2025 softcap record, a mean 0.001 below target needs 23 runs. A mean 0.002 below needs 9. Records are kept about 0.001 to 0.002 below the target on purpose, so a change that only buys loss buffer has to be converted into fewer steps.
Budget from those counts. One Track 2 run is 17.35 minutes on 8 GPUs, about 2.3 H100-hours, so 12 evidence runs cost about 28 H100-hours before any exploration. The Track 3 baseline is meant to take about 15 minutes and cost about MUV^\top$ from its SVD, computed by a few Newton-Schulz steps. Many recent records are tweaks to it.
Where people get stuck
- Cherry-picked runs. Every run you start counts. Dropping bad seeds invalidates the test, and Track 3 rules also forbid per-run early stopping based on validation loss.
- Hardware drift. Rented H100 nodes differ in clocks, interconnect and drivers. Always time the current record on the same node as your change, in the same session.
- Rules you didn't read. No changes to the data pipelines, and no extra
torch._inductor.configortorch.compileflags on Track 1. Track 2 allowscoordinate_descent_tuning. Track 3 forbids importing optimiser libraries: the code must be in the log. - Valid but not better. On Track 3 a result can reach the target in fewer steps and still not be significantly better than the previous one. The pairwise rule counts about 0.0045 loss per 100 steps when step counts differ.
- Compile time mistaken for speed. Warmup and compilation are outside the timed region. Profile steady-state steps, not the first ones.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- A merged pull request that sets a new record on any track, recorded with `python tools/mlarena.py record PL-W06 "<track> #<n>: <time or steps>"`.
- The evidence meets the track's rule with every run reported: p < 0.01 for Tracks 1 and 2, or (3.28 − mean)·√n ≥ 0.004 for Track 3.
- A profile-backed explanation of where the saved time or steps came from.
Side quests
Terms in plain words
- modded-nanogpt
- A public GitHub race to train a small GPT to a fixed validation loss in the least time on 8 GPUs.
- Speedrun
- A competition to reach a fixed target as fast as possible under fixed rules.
- p < 0.01
- A statistical test result meaning there is under a 1% chance the improvement came from run-to-run noise alone.
- Muon
- An optimiser that orthogonalises each weight matrix's momentum update; it came out of this speedrun.
- Profiling
- Measuring where time is spent on the GPU so you know which part is worth speeding up.
- CUDA graph
- A recorded sequence of GPU work that can be replayed with one launch, which removes per-kernel launch overhead.
- Pairwise significance
- Evidence that a new result beats the previous record itself, not only that it reaches the target.
Why it rates ★★★★★
A world record in the speedrun that Keller Jordan's work (including the Muon optimiser) came out of is about as strong a hiring signal as an individual can get.