← All Cracked Math projects
PM-W07NOT STARTEDWOWMATHPYTHONIMPACT ★★★★☆

Muon at the edge of stability: a sharpness study with your own autodiff

Gradient descent drives sharpness up to 2/eta, and Adam has its own threshold. Nobody has nailed down what Muon does, so measure it.

EST.65 h
REWARD+600 XP
AFTERCohen et al. 2021 + 3

ML-ARENA BENCHMARKS

PROOF: FINDING
METRICINDUSTRYMINE
Measured sharpness vs 2/eta (GD) and vs the preconditioned-sharpness threshold (Adam); Lanczos top-eigenvalue error vs a dense Hessian on a tiny model
GOAL · Reproduce GD EoS (sharpness within 10% of 2/eta on the plateau) and Adam's adaptive EoS; Lanczos eigenvalues within 1e-6 of the dense ones on the tiny model; a public write-up with code characterising Muon's sharpness dynamics
Cohen et al. 2021 (arXiv 2103.00065); Cohen et al. 2022 adaptive EoS (arXiv 2207.14484); PyHessian · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Sharpness is the largest eigenvalue of the loss Hessian, meaning the steepest curvature. Edge of stability (EoS) is the finding that with learning rate eta, full-batch gradient descent pushes sharpness up until it hovers at 2/eta, the point where classical theory says training should become unstable. Train a small transformer (1-5M parameters) in NumPy with your I11 autodiff. Every k steps, track the top Hessian eigenvalues via Lanczos (A06) on forward-over-reverse HVPs (Hessian-vector products, so the full Hessian is never built). Also track preconditioned sharpness for Adam (sharpness measured after Adam's per-parameter scaling). Reproduce edge-of-stability under full-batch GD and Adam's adaptive EoS, then characterise sharpness dynamics under Muon and under your A01 schedules.

VICTORY CONDITIONS

  1. Reproduce GD EoS (sharpness within 10% of 2/eta on the plateau) and Adam's adaptive EoS; Lanczos eigenvalues within 1e-6 of the dense ones on the tiny model; a public write-up with code characterising Muon's sharpness dynamics (Measured sharpness vs 2/eta (GD) and vs the preconditioned-sharpness threshold (Adam); Lanczos top-eigenvalue error vs a dense Hessian on a tiny model).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Sharpness
The largest eigenvalue of the loss Hessian, which measures the steepest curvature of the loss at the current weights.
Edge of stability (EoS)
The observed behaviour where gradient descent raises sharpness until it sits near 2 divided by the learning rate, yet training keeps making progress.
Preconditioned sharpness
The sharpness of the loss after rescaling by Adam's per-parameter step sizes, which is the quantity Adam's stability depends on.
Hessian-vector product (HVP)
The Hessian times a vector, computed with autodiff at about the cost of two gradients, without forming the Hessian.
Muon
An optimizer that orthogonalises each weight-matrix update with a Newton-Schulz polynomial (see A01).

Why it rates ★★★★☆

An original, carefully measured finding on a frontier optimizer's dynamics could be a workshop paper.