← All Cracked Math projects
PM-A01NOT STARTEDADVANCEDMATHPYTHONIMPACT ★★★☆☆

Newton-Schulz polynomials for the Muon optimizer: beat the default coefficients

Also known as: Newton-Schulz polynomials for Muon: beat the default coefficients

Muon orthogonalises each gradient matrix with a quintic (degree-5) polynomial. Treat it as a polynomial acting on singular values, and improve it in bf16.

EST.22 h
REWARD+400 XP
AFTERKeller Jordan + 1

ML-ARENA BENCHMARKS

PROOF: FINDING
METRICINDUSTRYMINE
max_i |sigma_i(X_5) - 1| (the worst distance of any singular value from 1 after 5 steps) and ||X_5^T X_5 - I||_F (how far X_5 is from orthogonal) in emulated bf16 on 50 real gradient matrices; validation loss of a toy run
GOAL · Reproduce both baselines' singular-value maps exactly. Find a 5-step schedule with lower worst-case |sigma - 1| than the default on real spectra. Confirm or refute a loss difference in a fixed-budget toy training run.
Muon default coefficients (Keller Jordan) and Polar Express (arXiv 2505.16932) · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Muon is an optimizer that replaces each weight-gradient matrix with the nearest orthogonal matrix, which means pushing all its singular values to 1. It does this cheaply with a Newton-Schulz iteration: apply an odd polynomial to the matrix several times. Analyse the odd-polynomial iteration p(x) = ax + bx^3 + cx^5 as a function acting on each singular value separately. Emulate bf16 (a 16-bit float format with the same range as float32 but far fewer mantissa bits, widely used on GPUs) in NumPy, by round-to-nearest-even on the float32 mantissa. Reproduce both Keller Jordan's quintic (3.4445, -4.7750, 2.0315) and Polar Express's per-iteration minimax schedule (different coefficients at each step, chosen to minimise the worst-case error). Then optimise a 5-step schedule on real gradient spectra (the singular values of real gradient matrices) collected while training a small model with your I11 autodiff, and test whether it changes the loss.

Approach

Start with the one fact that turns a matrix problem into a scalar one. Write the gradient as X=UΣV⊤X = U \Sigma V^\top. Then XX⊤X=UΣ3V⊤X X^\top X = U \Sigma^3 V^\top, and every odd polynomial keeps the same singular vectors:

aX+b (XX⊤)X+c (XX⊤)2X=U p(Σ) V⊤,p(σ)=aσ+bσ3+cσ5.aX + b\,(XX^\top)X + c\,(XX^\top)^2 X = U\,p(\Sigma)\,V^\top, \qquad p(\sigma) = a\sigma + b\sigma^3 + c\sigma^5 .

So five steps apply p5∘⋯∘p1p_5 \circ \dots \circ p_1 to each singular value on its own, and the whole question is how that scalar function behaves on [0,1][0, 1]. Dividing by the Frobenius norm first guarantees σmax⁡≤1\sigma_{\max} \le 1, because ∥X∥2≤∥X∥F\|X\|_2 \le \|X\|_F. The price is that small singular values start very small, often below 10−310^{-3}.

Keller Jordan's coefficients are not trying to converge. The slope at zero is a=3.4445a = 3.4445, so a tiny singular value grows by about 3.44455≈4853.4445^5 \approx 485 over five steps, and the map overshoots and oscillates around 1 instead of settling. Polar Express takes the other route: at step tt it picks the odd quintic that minimises max⁡σ∈[ℓt,ut]∣1−pt(σ)∣\max_{\sigma \in [\ell_t, u_t]} |1 - p_t(\sigma)|, and the next interval is the range of ptp_t on the current one. It starts from ℓ=10−3\ell = 10^{-3} for bf16 and replaces each pt(x)p_t(x) with pt(x/1.01)p_t(x / 1.01) as a safety margin against rounding.

Build in this order: the scalar map in float64, the bf16 emulator, the matrix iteration checked against the scalar map, then real spectra, then the optimiser. Measure the two benchmark numbers on held-out matrices before you touch training. The scalar map tells you what bf16 should do; the gap between them is the rounding story.

VISUALFig. 1
0 0.5 1 1e-4 1e-3 0.01 0.1 1 band: five steps land in 0.68 to 1.20 for σ above 0.003 (float64) input singular value σ (log scale, after Frobenius normalisation) output one step five steps
Keller Jordan's quintic in float64. One step (grey) pushes mid-range values toward 1 and barely moves small ones. Five steps (red) lift everything above about 0.003 into 0.68 to 1.20 and never settle there. Why it's here: The benchmark measures the worst point of this curve on your spectra, so you need to see where the worst points are.

Where people get stuck

  • Exact zeros. A weight gradient from a small batch is often rank deficient. Zero singular values stay zero under any odd polynomial, so max⁡i∣σi−1∣\max_i |\sigma_i - 1| is 1 for every schedule. Decide and state a floor, for example relative to σmax⁡\sigma_{\max}, and apply it to all three schedules.
  • Comparing different normalisations. Keller's code divides by ∥X∥F+10−7\|X\|_F + 10^{-7}. The Polar Express code in the paper's appendix divides by 1.01 ∥X∥F+10−71.01\,\|X\|_F + 10^{-7}. Copy each exactly from its source. Change the normalisation and you change the input interval, which can move the worst case more than the coefficients do.
  • Wrong bf16 emulation. Truncating the low 16 bits is round-toward-zero, not round-to-nearest-even, and it biases every product downward. Also decide where rounding happens: inputs only, or inputs and every matmul output. Write it down, because it changes the numbers.
  • Iterating on the big Gram matrix. Use XX⊤XX^\top with XX wide (transpose tall matrices first), as the reference code does. The result is the same in exact arithmetic but costs more and rounds differently.
  • Reading noise as signal. A loss difference smaller than the spread across seeds is not a finding. Fix the token budget and the learning-rate sweep before you look at the results.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

Objectives

0 / 7 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Both baselines reproduced, and a 5-step schedule with lower worst-case max |σ_i(X_5) − 1| than the default on held-out real gradient matrices in emulated bf16, recorded with `python tools/mlarena.py record PM-A01 "<yours> vs <default>"`.
  2. The orthogonality error ||X_5ᵀX_5 − I||_F reported for all three schedules on the same matrices.
  3. A fixed-budget toy training run with at least 3 seeds per schedule that confirms or refutes a validation-loss difference.

Side quests

Try degree-7 polynomials at the same matmul budget (count matmuls, not steps) and see whether fewer, higher-degree steps win.
Search for a schedule that targets the spectra you measured instead of a worst case over an interval: does a data-aware schedule beat Polar Express on loss, or only on the error metric?

Terms in plain words

Muon
An optimizer for neural-network weight matrices that orthogonalises each update, so every direction moves by a similar amount.
Newton-Schulz iteration
Repeatedly applying a matrix polynomial that drives all singular values toward 1, using only matrix multiplications.
Singular-value map
The scalar function p(x) applied to each singular value; because the polynomial is odd, the matrix iteration acts on singular values independently.
bf16
A 16-bit floating-point format with float32's exponent range but only 2 to 3 significant decimal digits.
Minimax schedule
Polynomial coefficients chosen to make the worst-case error over a range of inputs as small as possible.
Round-to-nearest-even
The default rounding rule in floating point. A value exactly halfway between two representable numbers goes to the one whose last bit is 0.

Why it rates ★★★☆☆

Frontier labs care about this kind of optimizer-internals literacy, but Polar Express already published the core idea.