← All Cracked Math projects
PM-I11NOT STARTEDINTERMEDIATEMATHPYTHONIMPACT ★★★☆☆

Tensor autodiff in both directions, with linear-algebra gradients

Not micrograd (the well-known tiny scalar autodiff tutorial): tensors, forward and reverse mode, Hessian-vector products, and the adjoints (backward rules) of inv, logdet and eigh.

EST.18 h
REWARD+300 XP
AFTERMax Slater + 2

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
Max relative gradient/HVP error vs PyTorch float64 on a 40-function suite; reverse-mode overhead vs PyTorch eager (how much slower your library runs)
GOAL · Gradients and HVPs match to 1e-10 on all 40 functions, with degenerate-eigenvalue cases (repeated eigenvalues, where the eigh gradient is undefined) detected and flagged. Reverse-mode overhead <= 3x PyTorch eager on a 3-layer MLP at batch 256.
PyTorch / JAX in float64 · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Automatic differentiation (autodiff) computes exact derivatives of code by applying the chain rule to each operation. Build a NumPy tensor autodiff library with: reverse mode (record operations on a tape, then sweep backwards, as backprop does); forward mode (dual tensors that carry a value and its derivative together); broadcasting-aware gradients (summing gradients back over the dimensions NumPy broadcast); JVPs (Jacobian-vector products, the forward-mode building block) and VJPs (vector-Jacobian products, the reverse-mode building block); and forward-over-reverse Hessian-vector products (run forward mode through the backward pass to get H times v without ever forming H). Add hand-derived backward rules for solve, inv, logdet, Cholesky and eigh, taken from Giles' collection of matrix derivatives. The constraint: tensor-valued from day one, and no autodiff library anywhere in the solution path.

VICTORY CONDITIONS

  1. Gradients and HVPs match to 1e-10 on all 40 functions, with degenerate-eigenvalue cases (repeated eigenvalues, where the eigh gradient is undefined) detected and flagged. Reverse-mode overhead <= 3x PyTorch eager on a 3-layer MLP at batch 256. (Max relative gradient/HVP error vs PyTorch float64 on a 40-function suite; reverse-mode overhead vs PyTorch eager (how much slower your library runs)).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Reverse vs forward mode
Reverse mode sweeps backwards from the output and gives all input gradients at once; forward mode carries derivatives along with values from the inputs.
JVP / VJP
Jacobian times a vector (forward mode) and vector times a Jacobian (reverse mode), computed without building the full Jacobian.
Hessian-vector product (HVP)
The Hessian matrix times a vector, computed by differentiating a gradient, at about the cost of two gradients.
Adjoint (backward rule)
The formula that turns the gradient with respect to an operation's output into the gradient with respect to its inputs.
eigh
The eigen-decomposition of a symmetric matrix; its gradient blows up when two eigenvalues are equal.

Why it rates ★★★☆☆

A tensor autodiff with correct matrix-function adjoints is a real systems-math signal; micrograd clones are not.