Brief
Sharpness is the largest eigenvalue of the loss Hessian, meaning the steepest curvature. Edge of stability (EoS) is the finding that with learning rate eta, full-batch gradient descent pushes sharpness up until it hovers at 2/eta, the point where classical theory says training should become unstable. Train a small transformer (1-5M parameters) in NumPy with your I11 autodiff. Every k steps, track the top Hessian eigenvalues via Lanczos (A06) on forward-over-reverse HVPs (Hessian-vector products, so the full Hessian is never built). Also track preconditioned sharpness for Adam (sharpness measured after Adam's per-parameter scaling). Reproduce edge-of-stability under full-batch GD and Adam's adaptive EoS, then characterise sharpness dynamics under Muon and under your A01 schedules.
VICTORY CONDITIONS
- Reproduce GD EoS (sharpness within 10% of 2/eta on the plateau) and Adam's adaptive EoS; Lanczos eigenvalues within 1e-6 of the dense ones on the tiny model; a public write-up with code characterising Muon's sharpness dynamics (Measured sharpness vs 2/eta (GD) and vs the preconditioned-sharpness threshold (Adam); Lanczos top-eigenvalue error vs a dense Hessian on a tiny model).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Sharpness
- The largest eigenvalue of the loss Hessian, which measures the steepest curvature of the loss at the current weights.
- Edge of stability (EoS)
- The observed behaviour where gradient descent raises sharpness until it sits near 2 divided by the learning rate, yet training keeps making progress.
- Preconditioned sharpness
- The sharpness of the loss after rescaling by Adam's per-parameter step sizes, which is the quantity Adam's stability depends on.
- Hessian-vector product (HVP)
- The Hessian times a vector, computed with autodiff at about the cost of two gradients, without forming the Hessian.
- Muon
- An optimizer that orthogonalises each weight-matrix update with a Newton-Schulz polynomial (see A01).
Why it rates ★★★★☆
An original, carefully measured finding on a frontier optimizer's dynamics could be a workshop paper.