Brief
There are three ways to get a derivative out of a computer. Symbolic algebra, which SymPy does, rewrites the formula, and the result can grow huge. Finite differences are easy, but rounding costs you a third to a half of your digits. Automatic differentiation is the third way, and every deep learning library runs on it.
The idea: any formula is built from a few elementary operations, and each has a known derivative. Apply the chain rule at every step and you get the exact derivative value, up to floating-point rounding. Forward mode carries a derivative along with each value. A neat way to do that is with dual numbers where , because then
Reverse mode goes the other way. First run the formula and record every operation. Then start at the output with and walk backwards. Each node passes its adjoint to its inputs, multiplied by the local derivative. When a value feeds several nodes, the contributions add up: , summed over every node that uses . One backward pass gives the derivative with respect to every input at once. That's backpropagation, and it's why training a model with millions of weights is affordable.
DATA · formulas you write
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- On 100 formulas, your forward- and reverse-mode derivatives both match SymPy within 1e-6 relative error at random points. Record the worst error with `python tools/mlarena.py record BM-17 "<worst rel error>"`.
- Reverse mode returns a 100-input gradient in one backward pass, and it agrees with forward mode.
- The README lists finite-difference results next to yours and notes where they lose digits.
Side quests
Terms in plain words
- Automatic differentiation (autodiff)
- Computing exact derivative values by applying the chain rule to every elementary operation as the formula runs. No symbolic algebra and no finite-difference step.
- Dual number
- A number a + bε with ε² = 0. Evaluating f at x + ε gives f(x) + f′(x)ε, so the derivative rides along with the value.
- Forward mode
- Pushes derivatives from the inputs toward the output. One pass per input variable.
- Reverse mode
- Records the computation, then pushes adjoints from the output back to the inputs. One backward pass gives the whole gradient. This is backpropagation.
- Adjoint
- For an intermediate value v, the derivative of the final output with respect to v, written v̄.