Brief
Stack 20 layers and plain training breaks. The loss sits flat, or it turns into NaN. In this project you build that broken network on purpose, measure why it fails, and fix it one piece at a time until it trains as well as the same setup in PyTorch.
The cause is scale. Each layer multiplies its input by a weight matrix. If the weights are a little too small, the signal shrinks at every layer and the gradients vanish. A little too large, and both explode. For a ReLU layer with inputs and weights of variance , the typical size of the signal changes by a factor of per layer. He initialisation picks so that the factor is exactly 1.
Initialisation only sets the scale at step zero. Normalisation keeps it in check during training. Layer norm rescales each example's activations to mean 0 and variance 1, then applies a learned scale and shift. Batch norm does the same for each feature across a batch. A residual connection, , gives the gradient a direct route back through the network.
Last comes the optimiser. Plain SGD takes the same step size for every weight. Momentum averages recent gradients
so the steps are smoother. Adam also divides each step by a running estimate of that weight's gradient size. Use
Fashion-MNIST: 70,000 grayscale 28 × 28 images of clothing in 10 classes. Load it with
torchvision.datasets.FashionMNIST or from OpenML with scikit-learn's fetch_openml.
DATA · a small public dataset
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your final training loss is within ±2% of the PyTorch twin on the same setup, recorded with `python tools/mlarena.py record BL-12 "<your loss> vs <torch loss>"`.
- A before-and-after chart of per-layer activation std and gradient norm for the broken network and the fixed one.
- Your layer norm backward pass agrees with finite differences to a relative error under 1e-6.
Side quests
Terms in plain words
- Vanishing gradient
- When the gradient shrinks a little at every layer on the way back, so the early layers barely change and the network stops learning.
- He initialisation
- Drawing a ReLU layer's starting weights with variance 2 divided by its number of inputs, so the signal keeps the same size from layer to layer.
- Layer normalisation
- Rescaling each example's activations to mean 0 and variance 1 inside a layer, then applying a learned scale and shift.
- Residual connection
- Adding a block's input to its output, so the signal and the gradient always have a direct path around the block.
- Adam
- An optimiser that keeps running averages of each weight's gradient and squared gradient, giving every weight its own step size.