Brief
Automatic differentiation (autodiff) computes exact derivatives of code by applying the chain rule to each operation. Build a NumPy tensor autodiff library with: reverse mode (record operations on a tape, then sweep backwards, as backprop does); forward mode (dual tensors that carry a value and its derivative together); broadcasting-aware gradients (summing gradients back over the dimensions NumPy broadcast); JVPs (Jacobian-vector products, the forward-mode building block) and VJPs (vector-Jacobian products, the reverse-mode building block); and forward-over-reverse Hessian-vector products (run forward mode through the backward pass to get H times v without ever forming H). Add hand-derived backward rules for solve, inv, logdet, Cholesky and eigh, taken from Giles' collection of matrix derivatives. The constraint: tensor-valued from day one, and no autodiff library anywhere in the solution path.
VICTORY CONDITIONS
- Gradients and HVPs match to 1e-10 on all 40 functions, with degenerate-eigenvalue cases (repeated eigenvalues, where the eigh gradient is undefined) detected and flagged. Reverse-mode overhead <= 3x PyTorch eager on a 3-layer MLP at batch 256. (Max relative gradient/HVP error vs PyTorch float64 on a 40-function suite; reverse-mode overhead vs PyTorch eager (how much slower your library runs)).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Reverse vs forward mode
- Reverse mode sweeps backwards from the output and gives all input gradients at once; forward mode carries derivatives along with values from the inputs.
- JVP / VJP
- Jacobian times a vector (forward mode) and vector times a Jacobian (reverse mode), computed without building the full Jacobian.
- Hessian-vector product (HVP)
- The Hessian matrix times a vector, computed by differentiating a gradient, at about the cost of two gradients.
- Adjoint (backward rule)
- The formula that turns the gradient with respect to an operation's output into the gradient with respect to its inputs.
- eigh
- The eigen-decomposition of a symmetric matrix; its gradient blows up when two eigenvalues are equal.
Why it rates ★★★☆☆
A tensor autodiff with correct matrix-function adjoints is a real systems-math signal; micrograd clones are not.