Brief
A neural ODE defines a model as the solution of a differential equation, so training it needs gradients through an ODE solver. Implement adaptive Dormand-Prince RK45 (a Runge-Kutta solver that estimates its own error and adjusts the step size) with error control and dense output (cheap interpolation between steps). Implement the continuous adjoint method: solve an augmented ODE backwards in time to get gradients without storing the forward steps, using VJPs from your I11 autodiff. Also implement discretise-then-optimise backprop through the solver: differentiate the solver's actual arithmetic steps, which costs memory for every step but is exact for the computed solution. Compare gradient accuracy and memory on spirals, Lotka-Volterra parameter fitting (predator-prey equations) and the chaotic Lorenz system.
Approach
The model is and the loss depends on . Define the adjoint . Chen et al. show that it obeys its own ODE, and that the parameter gradient is an integral along the way back:
Solve the stacked state backwards from , starting at . Both products come from one VJP of , so each backward step costs a small multiple of a forward one, and nothing from the forward pass is stored. That is the memory.
The catch is in the first component. The backward pass does not remember ; it rebuilds it by integrating the same ODE in reverse. A system that contracts going forward expands going backward. Lorenz with the usual parameters (, , ) has Lyapunov exponents of about , and . Going backward, the direction amplifies every rounding and truncation error by roughly , so the rebuilt is wrong long before the gradient itself is large. Separately, the true gradient grows like , which no solver can avoid.
For step control, Dormand-Prince gives a fifth-order answer and a fourth-order one from the same seven stages. Accept a step when the scaled error . A common rule for the next step is . Build the solver, then the reference gradient (backprop through the solver, checked by finite differences), then the adjoint, and compare every adjoint result with that reference.
Where people get stuck
- Comparing against the wrong truth. The adjoint gives the gradient of the exact ODE solution; backprop through the solver gives the gradient of what the solver computed. At loose tolerances they differ even on easy problems. Fix the tolerance before calling either one wrong.
- Different steps backwards. The backward solve picks its own steps, controlled by the error in and as well as . Set its tolerances on purpose, and report them.
- Float32 by accident. torchdiffeq follows the dtype of its inputs. Make every tensor float64, or a 1e-6 match is impossible.
- Finite differences in a chaotic system. On long Lorenz horizons, a perturbation of 1e-6 grows past the linear regime, so finite differences are wrong too. Keep finite-difference checks to short horizons.
- Memory that is not really flat. Storing the dense-output coefficients or the autodiff graph by accident keeps memory growing with steps. Measure it, don't assume.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Adjoint gradients within 1e-6 of torchdiffeq on the two non-chaotic tasks, with memory flat in step count, and a quantified growth of the adjoint gradient error against horizon on Lorenz. Record with `python tools/mlarena.py record PM-A05 "<worst rel err>, Lorenz rate <r>"`.
- Backprop-through-solver gradients pass a finite-difference check, so they can serve as the reference.
- The Lorenz growth rate is explained in terms of the Lyapunov spectrum, not just reported.
Side quests
Terms in plain words
- Neural ODE
- A model whose output is the solution of a differential equation whose right-hand side is a neural network.
- Adjoint method
- Computing gradients by solving a second ODE backwards in time, so memory does not grow with the number of solver steps.
- Dormand-Prince RK45
- A Runge-Kutta method that computes 4th- and 5th-order answers together and uses their difference to choose the step size.
- Discretise-then-optimise
- Backpropagating through the exact arithmetic the solver performed, which gives the true gradient of the computed answer but stores every step.
- Chaotic system
- A system where tiny changes in the start grow exponentially, which makes backward-in-time gradient computations unreliable.
- Lyapunov exponent
- The average exponential rate at which nearby trajectories separate (positive) or converge (negative). For the standard Lorenz system the largest is about 0.906.
- VJP (vector-Jacobian product)
- The product of a row vector with a Jacobian, which reverse-mode autodiff computes without forming the Jacobian.
Why it rates ★★★☆☆
Builds a known method from scratch. The Lorenz failure analysis is what shows judgement.