← All Base Form Math projects
BM-15NOT STARTEDBASE FORM · CALCULUSMATHPYTHON

Optimizer race

Animate gradient descent, momentum and Adam racing down tricky loss surfaces, and see why learning rate matters.

EST.6 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
steps to converge on Rosenbrock
GOAL · Adam within 10× BFGS steps; animation
scipy.optimize.minimize (BFGS) · measured—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Training a neural network means finding a low point on a loss surface. Gradient descent is the basic move. The gradient points uphill, so you step the other way:

xk+1=xk−η ∇f(xk)x_{k+1} = x_k - \eta\, \nabla f(x_k)

The learning rate η\eta is the hard part. In a long, narrow valley, a step big enough to make progress along the floor overshoots across the steep walls. Gradient descent zig-zags. If η\eta is larger than 2 divided by the steepest curvature, the zig-zag grows instead of shrinking, and the run blows up.

Momentum and Adam are two fixes. Momentum keeps a velocity made of past gradients, so the side-to-side parts cancel and the downhill part builds up. Adam keeps two running averages per coordinate, of the gradient gg and of its square: m←β1m+(1−β1)gm \leftarrow \beta_1 m + (1-\beta_1) g and v←β2v+(1−β2)g2v \leftarrow \beta_2 v + (1-\beta_2) g^2. Both start at zero, so it divides them by 1−βt1-\beta^t to correct the bias, and then steps by η m^/(v^+ϵ)\eta\, \hat m / (\sqrt{\hat v} + \epsilon). Coordinates with big gradients get smaller steps, and coordinates with small gradients get bigger ones.

The race track is the Rosenbrock function, a curved, banana-shaped valley with its minimum at (1, 1). BFGS, a method that builds up curvature information as it goes, gets there in about 30 iterations. Your goal is Adam within 10 times that. With the default settings Adam won't make it, so tuning is part of the project.

VISUALFig. 1
START plain GD: zig-zagsacross the steep walls momentum: the zig-zagscancel, the downhillpush adds up same 25 steps each
The same 25 steps on a narrow bowl. Plain gradient descent zig-zags; momentum takes a smoother path. Why it's here: This is the behaviour objectives 2 and 3 ask you to reproduce and explain.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

DATA · test functions

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. From (−1.2, 1) on Rosenbrock, Adam gets within 1e-4 of (1, 1) in at most 10 times the iterations BFGS reports in `res.nit` (32 when we ran it). Record both with `python tools/mlarena.py record BM-15 "<Adam steps> vs <BFGS nit>"`.
  2. The README has a GIF of all optimizers racing on the same contour plot.
  3. A table of steps for GD, momentum and Adam, with the settings used for each.

Side quests

Add a learning-rate schedule (step decay or cosine) and measure how much it helps Adam on Rosenbrock.
Add noise to the gradient to mimic mini-batches. Which optimizer copes best?
Add RMSProp or Nesterov momentum to the race.

Terms in plain words

Gradient descent
Repeatedly step against the gradient: x ← x − η∇f(x).
Learning rate
The step size η. Too small and it crawls; too large and it overshoots or blows up.
Momentum
Keeps a running velocity built from past gradients. Steps speed up along directions that stay the same and cancel along directions that keep flipping.
Adam
Scales each coordinate's step by a running estimate of that gradient's size, adds momentum, and corrects both averages for their zero start.
Learning-rate schedule
A rule that changes η during the run, often starting large and decaying.