Brief
Training a neural network means finding a low point on a loss surface. Gradient descent is the basic move. The gradient points uphill, so you step the other way:
The learning rate is the hard part. In a long, narrow valley, a step big enough to make progress along the floor overshoots across the steep walls. Gradient descent zig-zags. If is larger than 2 divided by the steepest curvature, the zig-zag grows instead of shrinking, and the run blows up.
Momentum and Adam are two fixes. Momentum keeps a velocity made of past gradients, so the side-to-side parts cancel and the downhill part builds up. Adam keeps two running averages per coordinate, of the gradient and of its square: and . Both start at zero, so it divides them by to correct the bias, and then steps by . Coordinates with big gradients get smaller steps, and coordinates with small gradients get bigger ones.
The race track is the Rosenbrock function, a curved, banana-shaped valley with its minimum at (1, 1). BFGS, a method that builds up curvature information as it goes, gets there in about 30 iterations. Your goal is Adam within 10 times that. With the default settings Adam won't make it, so tuning is part of the project.
DATA · test functions
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- From (−1.2, 1) on Rosenbrock, Adam gets within 1e-4 of (1, 1) in at most 10 times the iterations BFGS reports in `res.nit` (32 when we ran it). Record both with `python tools/mlarena.py record BM-15 "<Adam steps> vs <BFGS nit>"`.
- The README has a GIF of all optimizers racing on the same contour plot.
- A table of steps for GD, momentum and Adam, with the settings used for each.
Side quests
Terms in plain words
- Gradient descent
- Repeatedly step against the gradient: x ← x − η∇f(x).
- Learning rate
- The step size η. Too small and it crawls; too large and it overshoots or blows up.
- Momentum
- Keeps a running velocity built from past gradients. Steps speed up along directions that stay the same and cancel along directions that keep flipping.
- Adam
- Scales each coordinate's step by a running estimate of that gradient's size, adds momentum, and corrects both averages for their zero start.
- Learning-rate schedule
- A rule that changes η during the run, often starting large and decaying.