← All Cracked Math projects
PM-A05NOT STARTEDADVANCEDMATHPYTHONIMPACT ★★★☆☆

Neural-ODE adjoint from scratch, and where it lies

Getting gradients through an ODE solver with O(1) memory is elegant. Find out exactly when those gradients are wrong.

EST.25 h
REWARD+400 XP
AFTERChen et al. 2018 + 1

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
Gradient relative error vs backprop-through-solver at matched tolerance; peak memory vs number of steps
GOAL · Adjoint gradients within 1e-6 of torchdiffeq on the 2 non-chaotic tasks, with memory flat in step count; quantified growth of the adjoint gradient error vs time horizon on Lorenz
torchdiffeq odeint_adjoint (Chen et al. 2018) · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

A neural ODE defines a model as the solution of a differential equation, so training it needs gradients through an ODE solver. Implement adaptive Dormand-Prince RK45 (a Runge-Kutta solver that estimates its own error and adjusts the step size) with error control and dense output (cheap interpolation between steps). Implement the continuous adjoint method: solve an augmented ODE backwards in time to get gradients without storing the forward steps, using VJPs from your I11 autodiff. Also implement discretise-then-optimise backprop through the solver: differentiate the solver's actual arithmetic steps, which costs memory for every step but is exact for the computed solution. Compare gradient accuracy and memory on spirals, Lotka-Volterra parameter fitting (predator-prey equations) and the chaotic Lorenz system.

Approach

The model is z˙=f(z,t,θ)\dot z = f(z, t, \theta) and the loss LL depends on z(t1)z(t_1). Define the adjoint a(t)=∂L/∂z(t)a(t) = \partial L / \partial z(t). Chen et al. show that it obeys its own ODE, and that the parameter gradient is an integral along the way back:

dadt=−a(t)⊤∂f∂z,dLdθ=−∫t1t0a(t)⊤∂f∂θ dt.\frac{da}{dt} = -a(t)^\top \frac{\partial f}{\partial z}, \qquad \frac{dL}{d\theta} = -\int_{t_1}^{t_0} a(t)^\top \frac{\partial f}{\partial \theta}\, dt .

Solve the stacked state [z,a,∂L/∂θ][z, a, \partial L/\partial\theta] backwards from t1t_1, starting at [z(t1),∂L/∂z(t1),0][z(t_1), \partial L/\partial z(t_1), 0]. Both products come from one VJP of ff, so each backward step costs a small multiple of a forward one, and nothing from the forward pass is stored. That is the O(1)O(1) memory.

The catch is in the first component. The backward pass does not remember z(t)z(t); it rebuilds it by integrating the same ODE in reverse. A system that contracts going forward expands going backward. Lorenz with the usual parameters (σ=10\sigma = 10, ρ=28\rho = 28, β=8/3\beta = 8/3) has Lyapunov exponents of about 0.9060.906, 00 and −14.57-14.57. Going backward, the −14.57-14.57 direction amplifies every rounding and truncation error by roughly e14.57Te^{14.57 T}, so the rebuilt zz is wrong long before the gradient itself is large. Separately, the true gradient grows like e0.906Te^{0.906 T}, which no solver can avoid.

For step control, Dormand-Prince gives a fifth-order answer and a fourth-order one from the same seven stages. Accept a step when the scaled error err=∥(y5−y^4)/(atol+rtol ∣y∣)∥RMS≤1\text{err} = \big\| (y_5 - \hat y_4) / (\text{atol} + \text{rtol}\,|y|) \big\|_{\text{RMS}} \le 1. A common rule for the next step is h⋅min⁡(5,max⁡(0.2,0.9 err−1/5))h \cdot \min(5, \max(0.2, 0.9\,\text{err}^{-1/5})). Build the solver, then the reference gradient (backprop through the solver, checked by finite differences), then the adjoint, and compare every adjoint result with that reference.

VISUALFig. 1
1e-12 1e-8 1e-4 1e0 0 1 2 3 backward rebuild of z(0), relative error then blows up error above 1e-2 horizon T 1e0 1e4 1e8 1e12 0 10 20 30 ‖∂z(T)/∂z(0)‖, the true gradient size dashed: e^(0.906 T) horizon T
Lorenz from a point on the attractor, fixed-step RK4 in float64. Left: run forward to T, then backward, and compare with the start. Right: the norm of the true sensitivity matrix from the variational equation, with the largest Lyapunov exponent as a reference line. Why it's here: Two different failures with two different rates. Your Lorenz plot in objective 6 should show which one dominates the adjoint's error.

Where people get stuck

  • Comparing against the wrong truth. The adjoint gives the gradient of the exact ODE solution; backprop through the solver gives the gradient of what the solver computed. At loose tolerances they differ even on easy problems. Fix the tolerance before calling either one wrong.
  • Different steps backwards. The backward solve picks its own steps, controlled by the error in aa and ∂L/∂θ\partial L/\partial \theta as well as zz. Set its tolerances on purpose, and report them.
  • Float32 by accident. torchdiffeq follows the dtype of its inputs. Make every tensor float64, or a 1e-6 match is impossible.
  • Finite differences in a chaotic system. On long Lorenz horizons, a perturbation of 1e-6 grows past the linear regime, so finite differences are wrong too. Keep finite-difference checks to short horizons.
  • Memory that is not really flat. Storing the dense-output coefficients or the autodiff graph by accident keeps memory growing with steps. Measure it, don't assume.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 6
Stuck? Hints stay hidden until you ask.

Objectives

0 / 7 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Adjoint gradients within 1e-6 of torchdiffeq on the two non-chaotic tasks, with memory flat in step count, and a quantified growth of the adjoint gradient error against horizon on Lorenz. Record with `python tools/mlarena.py record PM-A05 "<worst rel err>, Lorenz rate <r>"`.
  2. Backprop-through-solver gradients pass a finite-difference check, so they can serve as the reference.
  3. The Lorenz growth rate is explained in terms of the Lyapunov spectrum, not just reported.

Side quests

Add checkpointing (store a few states on the way forward and re-solve each segment backwards from its checkpoint) and measure how far it pushes the Lorenz horizon.
Read Kidger, Chen and Lyons 2021, "Hey, that's not an ODE", and try their seminorm step control for the backward pass. Count the steps it saves.

Terms in plain words

Neural ODE
A model whose output is the solution of a differential equation whose right-hand side is a neural network.
Adjoint method
Computing gradients by solving a second ODE backwards in time, so memory does not grow with the number of solver steps.
Dormand-Prince RK45
A Runge-Kutta method that computes 4th- and 5th-order answers together and uses their difference to choose the step size.
Discretise-then-optimise
Backpropagating through the exact arithmetic the solver performed, which gives the true gradient of the computed answer but stores every step.
Chaotic system
A system where tiny changes in the start grow exponentially, which makes backward-in-time gradient computations unreliable.
Lyapunov exponent
The average exponential rate at which nearby trajectories separate (positive) or converge (negative). For the standard Lorenz system the largest is about 0.906.
VJP (vector-Jacobian product)
The product of a row vector with a Jacobian, which reverse-mode autodiff computes without forming the Jacobian.

Why it rates ★★★☆☆

Builds a known method from scratch. The Lorenz failure analysis is what shows judgement.