Brief
In NumPy, build reverse-mode autodiff over n-d arrays. You need a topologically sorted tape (a record of the operations, replayed backwards in the right order), broadcast-aware unbroadcast reductions (when an input was broadcast to a bigger shape, sum its gradient back down to the original shape), views and strides (tensors that share memory but have a different shape), and 40 ops including matmul, softmax, log-softmax cross-entropy, LayerNorm, BatchNorm, im2col conv (convolution rewritten as one big matmul) and gather. Use it for an init and normalisation study: a 50-layer plain MLP fails to train, while He init + LayerNorm + residuals trains.
VICTORY CONDITIONS
- relative error < 1e-6 on all ops; ≥ 50% of PyTorch eager throughput; the depth-50 residual+LN net trains to the same loss as a depth-5 net (max relative gradient error vs torch.autograd (fp64) over 40 ops × random shapes; MLP train-step time vs PyTorch CPU eager mode; whether the depth-50 net trains).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Reverse-mode autodiff
- Recording a computation and then walking it backwards to get the gradient of one output with respect to every input.
- Tape
- The ordered list of operations recorded during the forward pass, used to run the backward pass.
- Unbroadcast
- Summing a gradient over the axes that broadcasting stretched, so it matches the original input's shape.
- im2col
- Copying image patches into columns of a matrix so a convolution becomes a single matrix multiply.
- He init
- Random starting weights scaled by the square root of 2 over the layer's input size, which keeps signals stable through ReLU layers.
- Eager mode
- PyTorch's normal mode, where each operation runs immediately with no compilation.
Why it rates ★★☆☆☆
A tensor-level engine beats the overdone scalar micrograd, but this is still a learning artefact.