← All Cracked ML projects
PL-I04NOT STARTEDINTERMEDIATEMLPYTHONIMPACT ★★☆☆☆

Tensor autograd engine, gradient-checked against PyTorch

Not micrograd: a tensor autograd engine that understands broadcasting, with 40 ops, every one checked against torch.autograd.

EST.16 h
REWARD+300 XP
AFTERMax Slater + 2

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
max relative gradient error vs torch.autograd (fp64) over 40 ops × random shapes; MLP train-step time vs PyTorch CPU eager mode; whether the depth-50 net trains
GOAL · relative error < 1e-6 on all ops; ≥ 50% of PyTorch eager throughput; the depth-50 residual+LN net trains to the same loss as a depth-5 net
PyTorch autograd · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In NumPy, build reverse-mode autodiff over n-d arrays. You need a topologically sorted tape (a record of the operations, replayed backwards in the right order), broadcast-aware unbroadcast reductions (when an input was broadcast to a bigger shape, sum its gradient back down to the original shape), views and strides (tensors that share memory but have a different shape), and 40 ops including matmul, softmax, log-softmax cross-entropy, LayerNorm, BatchNorm, im2col conv (convolution rewritten as one big matmul) and gather. Use it for an init and normalisation study: a 50-layer plain MLP fails to train, while He init + LayerNorm + residuals trains.

VICTORY CONDITIONS

  1. relative error < 1e-6 on all ops; ≥ 50% of PyTorch eager throughput; the depth-50 residual+LN net trains to the same loss as a depth-5 net (max relative gradient error vs torch.autograd (fp64) over 40 ops × random shapes; MLP train-step time vs PyTorch CPU eager mode; whether the depth-50 net trains).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Reverse-mode autodiff
Recording a computation and then walking it backwards to get the gradient of one output with respect to every input.
Tape
The ordered list of operations recorded during the forward pass, used to run the backward pass.
Unbroadcast
Summing a gradient over the axes that broadcasting stretched, so it matches the original input's shape.
im2col
Copying image patches into columns of a matrix so a convolution becomes a single matrix multiply.
He init
Random starting weights scaled by the square root of 2 over the layer's input size, which keeps signals stable through ReLU layers.
Eager mode
PyTorch's normal mode, where each operation runs immediately with no compilation.

Why it rates ★★☆☆☆

A tensor-level engine beats the overdone scalar micrograd, but this is still a learning artefact.