← All Base Form ML projects
BL-11NOT STARTEDBASE FORM · DEEP LEARNINGMLPYTHON

Neural net from scratch

An MLP in NumPy that learns to separate spirals, with a live decision boundary while it trains.

EST.6 h
REWARD+100 XP
AFTERAndrej Karpathy

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
gradients; accuracy
GOAL · gradients within 1e-6; ≥ 99% on spirals
finite differences—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

You build a small neural network in NumPy and teach it to sort points on three interleaved spirals. There is no autograd library here. You write the forward pass, work out every gradient by hand with the chain rule, and prove the gradients are right before you trust them.

A multilayer perceptron (MLP) is a stack of linear layers with a nonlinear activation between them. One hidden layer is enough for spirals: h=ReLU(XW1+b1)h = \mathrm{ReLU}(XW_1 + b_1), then class scores z=hW2+b2z = hW_2 + b_2, then softmax and cross-entropy, the same loss as your softmax regression. Softmax regression on its own can only draw straight boundaries, so it fails here. The hidden layer is what lets the network bend them.

Backpropagation is the chain rule applied layer by layer, from the loss back to the inputs. Each layer receives the gradient of the loss with respect to its output and returns the gradient with respect to its input and its weights. For softmax with cross-entropy the first step is short: ∂L/∂z=p−y\partial L / \partial z = p - y, the predicted probabilities minus the one-hot label, divided by the batch size if your loss is a mean.

To check a gradient, nudge one weight by a small hh and measure the change in loss: (L(θ+h)−L(θ−h))/2h\big(L(\theta + h) - L(\theta - h)\big) / 2h. This centred difference should match your analytic gradient closely. Generate the spirals yourself. The Stanford CS231n notes ("Minimal Neural Network Case Study") give the standard few-line recipe. Then watch the boundary curl around the arms as the network trains.

VISUALFig. 1
FORWARD → X LINEAR 1a = XW₁ + b₁ RELUh = max(0, a) LINEAR 2z = hW₂ + b₂ SOFTMAX + CEp, then loss L dz = p − y dW₂, db₂, dh da = dh · [a > 0] dW₁, db₁ ← BACKWARD
The network as a chain of steps. The forward pass runs left to right; backpropagation sends gradients right to left through the same steps. Why it's here: Objective 3 is easier when you see that each box needs only its own local rule for the backward pass.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.

DATA · generated spirals

Objectives

0 / 5 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Largest relative gradient error below 1e-6 and at least 99% training accuracy on the spirals, recorded with `python tools/mlarena.py record BL-11 "<max rel error>; <accuracy>"`.
  2. NumPy only. No autograd library, and every gradient is written by hand.
  3. The README shows the final decision boundary and the loss curve.

Side quests

Draw a second spiral sample with a different seed and report accuracy on it. Then shrink the hidden layer until that number drops.
Turn your layers into a tiny autograd engine in the style of micrograd, where each operation knows its own backward step.

Terms in plain words

Multilayer perceptron (MLP)
A stack of linear layers with a nonlinear function between each pair, so the whole thing can bend its decision boundaries.
Activation function
The nonlinear function applied after a linear layer. ReLU, max(0, x), is the usual default.
Backpropagation
The chain rule run backwards through the network, turning the gradient of the loss into a gradient for every weight.
Finite difference
An estimate of a gradient made by nudging one number up and down a tiny bit and measuring how much the loss changes.
Decision boundary
The line or curve in input space where the model switches from predicting one class to another.