Brief
Muon is an optimizer that replaces each weight-gradient matrix with the nearest orthogonal matrix, which means pushing all its singular values to 1. It does this cheaply with a Newton-Schulz iteration: apply an odd polynomial to the matrix several times. Analyse the odd-polynomial iteration p(x) = ax + bx^3 + cx^5 as a function acting on each singular value separately. Emulate bf16 (a 16-bit float format with the same range as float32 but far fewer mantissa bits, widely used on GPUs) in NumPy, by round-to-nearest-even on the float32 mantissa. Reproduce both Keller Jordan's quintic (3.4445, -4.7750, 2.0315) and Polar Express's per-iteration minimax schedule (different coefficients at each step, chosen to minimise the worst-case error). Then optimise a 5-step schedule on real gradient spectra (the singular values of real gradient matrices) collected while training a small model with your I11 autodiff, and test whether it changes the loss.
Approach
Start with the one fact that turns a matrix problem into a scalar one. Write the gradient as . Then , and every odd polynomial keeps the same singular vectors:
So five steps apply to each singular value on its own, and the whole question is how that scalar function behaves on . Dividing by the Frobenius norm first guarantees , because . The price is that small singular values start very small, often below .
Keller Jordan's coefficients are not trying to converge. The slope at zero is , so a tiny singular value grows by about over five steps, and the map overshoots and oscillates around 1 instead of settling. Polar Express takes the other route: at step it picks the odd quintic that minimises , and the next interval is the range of on the current one. It starts from for bf16 and replaces each with as a safety margin against rounding.
Build in this order: the scalar map in float64, the bf16 emulator, the matrix iteration checked against the scalar map, then real spectra, then the optimiser. Measure the two benchmark numbers on held-out matrices before you touch training. The scalar map tells you what bf16 should do; the gap between them is the rounding story.
Where people get stuck
- Exact zeros. A weight gradient from a small batch is often rank deficient. Zero singular values stay zero under any odd polynomial, so is 1 for every schedule. Decide and state a floor, for example relative to , and apply it to all three schedules.
- Comparing different normalisations. Keller's code divides by . The Polar Express code in the paper's appendix divides by . Copy each exactly from its source. Change the normalisation and you change the input interval, which can move the worst case more than the coefficients do.
- Wrong bf16 emulation. Truncating the low 16 bits is round-toward-zero, not round-to-nearest-even, and it biases every product downward. Also decide where rounding happens: inputs only, or inputs and every matmul output. Write it down, because it changes the numbers.
- Iterating on the big Gram matrix. Use with wide (transpose tall matrices first), as the reference code does. The result is the same in exact arithmetic but costs more and rounds differently.
- Reading noise as signal. A loss difference smaller than the spread across seeds is not a finding. Fix the token budget and the learning-rate sweep before you look at the results.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Both baselines reproduced, and a 5-step schedule with lower worst-case max |σ_i(X_5) − 1| than the default on held-out real gradient matrices in emulated bf16, recorded with `python tools/mlarena.py record PM-A01 "<yours> vs <default>"`.
- The orthogonality error ||X_5ᵀX_5 − I||_F reported for all three schedules on the same matrices.
- A fixed-budget toy training run with at least 3 seeds per schedule that confirms or refutes a validation-loss difference.
Side quests
Terms in plain words
- Muon
- An optimizer for neural-network weight matrices that orthogonalises each update, so every direction moves by a similar amount.
- Newton-Schulz iteration
- Repeatedly applying a matrix polynomial that drives all singular values toward 1, using only matrix multiplications.
- Singular-value map
- The scalar function p(x) applied to each singular value; because the polynomial is odd, the matrix iteration acts on singular values independently.
- bf16
- A 16-bit floating-point format with float32's exponent range but only 2 to 3 significant decimal digits.
- Minimax schedule
- Polynomial coefficients chosen to make the worst-case error over a range of inputs as small as possible.
- Round-to-nearest-even
- The default rounding rule in floating point. A value exactly halfway between two representable numbers goes to the one whose last bit is 0.
Why it rates ★★★☆☆
Frontier labs care about this kind of optimizer-internals literacy, but Polar Express already published the core idea.