← All Base Form ML projects
BL-12NOT STARTEDBASE FORM · DEEP LEARNINGMLPYTHON

Deep-net rescue

Train a 20-layer network that refuses to learn, then fix it with good initialisation, normalisation and a better optimiser.

EST.7 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
training curve
GOAL · same final loss ± 2%
PyTorch, same setup—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Stack 20 layers and plain training breaks. The loss sits flat, or it turns into NaN. In this project you build that broken network on purpose, measure why it fails, and fix it one piece at a time until it trains as well as the same setup in PyTorch.

The cause is scale. Each layer multiplies its input by a weight matrix. If the weights are a little too small, the signal shrinks at every layer and the gradients vanish. A little too large, and both explode. For a ReLU layer with nn inputs and weights of variance σ2\sigma^2, the typical size of the signal changes by a factor of nσ2/2\sqrt{n\sigma^2/2} per layer. He initialisation picks σ2=2/n\sigma^2 = 2/n so that the factor is exactly 1.

Initialisation only sets the scale at step zero. Normalisation keeps it in check during training. Layer norm rescales each example's activations to mean 0 and variance 1, then applies a learned scale and shift. Batch norm does the same for each feature across a batch. A residual connection, xl+1=xl+f(xl)x_{l+1} = x_l + f(x_l), gives the gradient a direct route back through the network.

Last comes the optimiser. Plain SGD takes the same step size for every weight. Momentum averages recent gradients so the steps are smoother. Adam also divides each step by a running estimate of that weight's gradient size. Use Fashion-MNIST: 70,000 grayscale 28 × 28 images of clothing in 10 classes. Load it with torchvision.datasets.FashionMNIST or from OpenML with scikit-learn's fetch_openml.

VISUALFig. 1
0.001 0.01 0.1 1 10 100 1000 0 5 10 15 20 layer activation size (log scale) σ² = 1/n σ² = 2/n (He) σ² = 4/n
Theory for a deep ReLU network: the typical activation size at each layer for three weight variances. Only He initialisation (2/n) keeps it level. Why it's here: Objectives 1 and 2 ask you to plot this from your own network. This is the shape to expect.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

DATA · a small public dataset

Objectives

0 / 5 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your final training loss is within ±2% of the PyTorch twin on the same setup, recorded with `python tools/mlarena.py record BL-12 "<your loss> vs <torch loss>"`.
  2. A before-and-after chart of per-layer activation std and gradient norm for the broken network and the fixed one.
  3. Your layer norm backward pass agrees with finite differences to a relative error under 1e-6.

Side quests

Swap layer norm for batch norm, then train with a batch size of 4. Explain what goes wrong.
Make an ablation table that removes one fix at a time and shows which one matters most at depth 20.
Read about the Muon optimiser, which orthogonalises the momentum update of each weight matrix, and try it on the hidden layers.

Terms in plain words

Vanishing gradient
When the gradient shrinks a little at every layer on the way back, so the early layers barely change and the network stops learning.
He initialisation
Drawing a ReLU layer's starting weights with variance 2 divided by its number of inputs, so the signal keeps the same size from layer to layer.
Layer normalisation
Rescaling each example's activations to mean 0 and variance 1 inside a layer, then applying a learned scale and shift.
Residual connection
Adding a block's input to its output, so the signal and the gradient always have a direct path around the block.
Adam
An optimiser that keeps running averages of each weight's gradient and squared gradient, giving every weight its own step size.