← All Cracked ML projects
PL-W06NOT STARTEDWOWMLPYTHONIMPACT ★★★★★

Set a modded-nanogpt world record

Get a PR merged as a world record in the most competitive training speedrun there is.

EST.80 h
REWARD+600 XP
AFTERKellerJordan/modded-nanogpt + 1

ML-ARENA BENCHMARKS

PROOF: UPSTREAM
METRICINDUSTRYMINE
wall-clock time (or steps, on Track 3) to reach the target val loss on 8×H100, with p < 0.01 over runs
GOAL · a merged PR that sets a new record on any track
modded-nanogpt records: Track 1 #92 at 0.665 min (2026-08-30); Track 2 #18 at 17.35 min (2025-12-31)—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Study the record history, profile the current record on 8×H100 (rented), and find a statistically significant improvement: an optimiser change (Muon variants), an architecture tweak, kernel fusion or a schedule change. Track 1 (GPT-2 124M → val loss 3.28) is now about 40 s, so the realistic targets are Track 2 (GPT-2 Medium → 2.92) or Track 3 (fewest steps). Rules: no data-pipeline changes, and p < 0.01 over multiple runs (the gain must be very unlikely to be luck).

Approach

Pick the track before anything else, because each has its own economics. Track 1 (GPT-2 Small to 3.28) stood at 0.665 minutes after record #92 on 2026-08-30. Its recent records come from FP8 kernels, sampled softmax, a hashed n-gram table and full CUDA-graph capture, so the remaining gains there are mostly systems work. Track 2 (GPT-2 Medium to 2.92) stood at 17.35 minutes after record #18 on 2025-12-31, with far fewer records so far. Track 3 fixes the architecture, data and batch size and counts steps to 3.28. Wall-clock time doesn't matter there, so it is an optimiser contest. Its best result was 2,690 steps (#46, 2026-06-19).

The statistics are part of the submission. For Tracks 1 and 2 you need a one-sided test that the mean loss over nn runs is below the target:

t=target−xˉs/n,p=P(Tn−1>t)<0.01.t = \frac{\text{target} - \bar{x}}{s / \sqrt{n}}, \qquad p = P(T_{n-1} > t) < 0.01 .

Track 3 uses a z-test with a fixed σ=0.0013\sigma = 0.0013, which reduces to (3.28−xˉ)n≥0.004(3.28 - \bar{x})\sqrt{n} \ge 0.004. The figure shows what that costs. With the run-to-run spread of 0.0019 measured over 80 runs in the January 2025 softcap record, a mean 0.001 below target needs 23 runs. A mean 0.002 below needs 9. Records are kept about 0.001 to 0.002 below the target on purpose, so a change that only buys loss buffer has to be converted into fewer steps.

Budget from those counts. One Track 2 run is 17.35 minutes on 8 GPUs, about 2.3 H100-hours, so 12 evidence runs cost about 28 H100-hours before any exploration. The Track 3 baseline is meant to take about 15 minutes and cost about 6perexperiment.ATrack1runisunderaminute,butthefirst‘torch.compile‘addsabout7minutes.Muon,theoptimisermostrecordsbuildon,replaceseachweightmatrix′smomentum6 per experiment. A Track 1 run is under a minute, but the first `torch.compile` adds about 7 minutes. Muon, the optimiser most records build on, replaces each weight matrix's momentum Mwithanapproximatewith an approximateUV^\top$ from its SVD, computed by a few Newton-Schulz steps. Many recent records are tweaks to it.

VISUALFig. 1
where records usually sit 1 2 5 10 20 50 100 0.0005 0.0010 0.0015 0.0020 0.0025 0.0030 0.0035 0.0040 margin below target: 3.28 minus mean val loss runs needed (log scale) 23 16 9 4 Tracks 1–2one-sided t-test,p < 0.01, σ = 0.0019 Track 3(3.28 − mean)·√n≥ 0.004
Runs needed to pass each track's significance rule, against how far the mean validation loss sits below 3.28. Tracks 1 and 2 use a one-sided t-test with the spread from an 80-run record; Track 3 uses its z-rule. Why it's here: The number of runs sets your rental bill, so work it out before you start collecting evidence.

Where people get stuck

  • Cherry-picked runs. Every run you start counts. Dropping bad seeds invalidates the test, and Track 3 rules also forbid per-run early stopping based on validation loss.
  • Hardware drift. Rented H100 nodes differ in clocks, interconnect and drivers. Always time the current record on the same node as your change, in the same session.
  • Rules you didn't read. No changes to the data pipelines, and no extra torch._inductor.config or torch.compile flags on Track 1. Track 2 allows coordinate_descent_tuning. Track 3 forbids importing optimiser libraries: the code must be in the log.
  • Valid but not better. On Track 3 a result can reach the target in fewer steps and still not be significantly better than the previous one. The pairwise rule counts about 0.0045 loss per 100 steps when step counts differ.
  • Compile time mistaken for speed. Warmup and compilation are outside the timed region. Profile steady-state steps, not the first ones.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

Objectives

0 / 8 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. A merged pull request that sets a new record on any track, recorded with `python tools/mlarena.py record PL-W06 "<track> #<n>: <time or steps>"`.
  2. The evidence meets the track's rule with every run reported: p < 0.01 for Tracks 1 and 2, or (3.28 − mean)·√n ≥ 0.004 for Track 3.
  3. A profile-backed explanation of where the saved time or steps came from.

Side quests

Port a Track 3 optimiser result to Track 1 or 2 and see whether fewer steps still means less wall-clock time once the optimiser's own cost is counted.
Submit a skeptical result: rerun an older record with more seeds and test whether its claim holds. The Track 3 README welcomes these.

Terms in plain words

modded-nanogpt
A public GitHub race to train a small GPT to a fixed validation loss in the least time on 8 GPUs.
Speedrun
A competition to reach a fixed target as fast as possible under fixed rules.
p < 0.01
A statistical test result meaning there is under a 1% chance the improvement came from run-to-run noise alone.
Muon
An optimiser that orthogonalises each weight matrix's momentum update; it came out of this speedrun.
Profiling
Measuring where time is spent on the GPU so you know which part is worth speeding up.
CUDA graph
A recorded sequence of GPU work that can be replayed with one launch, which removes per-kernel launch overhead.
Pairwise significance
Evidence that a new result beats the previous record itself, not only that it reaches the target.

Why it rates ★★★★★

A world record in the speedrun that Keller Jordan's work (including the Muon optimiser) came out of is about as strong a hiring signal as an individual can get.