← All Base Form ML projects
BL-21NOT STARTEDBASE FORM · TRANSFORMERSMLPYTHON

Tiny GPT

Build a small GPT from scratch and train it on your favourite book until it writes convincing paragraphs.

EST.12 h
REWARD+100 XP
AFTERAndrej Karpathy

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
validation loss
GOAL · within 5%
nanoGPT, same data and size—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

A GPT is a language model that predicts the next token from all the tokens before it. You built one in BL-15 with an LSTM, which squeezes the past into a single hidden state. A transformer does something different: at every position it looks back at all earlier positions directly and decides which ones matter. That mechanism is attention, and it is the new idea here.

Each position makes three vectors from its input: a query qq (what am I looking for), a key kk (what do I contain) and a value vv (what do I pass on). Stack them into matrices and one head computes

Attention(Q,K,V)=softmax ⁣(QK⊤dk+M)V\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V

where dkd_k is the key size and MM is the causal mask: 00 on and below the diagonal, −∞-\infty above it. Dividing by dk\sqrt{d_k} keeps the scores from growing with the vector size, so the softmax doesn't saturate. Several heads run in parallel and their outputs are joined back together. A block is attention plus a small MLP, each wrapped in LayerNorm and a residual connection, both of which you met in BL-12.

You'll write the model in PyTorch from nn.Linear, nn.Embedding and nn.LayerNorm up, without nn.MultiheadAttention or nn.Transformer. Keep the code tidy and save the checkpoint well. This model is the one you speed up in BL-22, serve in BL-27, look inside in BL-29 and steer in BL-30.

VISUALFig. 1
KEYS: POSITIONS IT CAN LOOK AT QUERIES the ␣ca the ␣ca allowed softmax over these; a row sums to 1 masked score set to −∞, weight becomes 0 one query "c" mixes t, h, e, ␣ and itself; it never sees the "a" it must predict
The causal mask: row i is the query at position i, and it may only attend to columns 0 to i. Why it's here: Objective 2 is mostly about getting this triangle right; see it before you code the mask.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.

DATA · a Project Gutenberg book

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your validation loss is within 5% of nanoGPT's on the same data, size and step count. Record both with `python tools/mlarena.py record BL-21 "<yours> vs <nanoGPT>"`.
  2. Your attention matches scaled_dot_product_attention to 1e-5 and passes the "future tokens change nothing" test.
  3. A 500-character sample from the final model reads like the book, and the saved checkpoint plus model code load cleanly in a fresh script (BL-22, BL-27, BL-29 and BL-30 all start from it).

Side quests

Replace learned position embeddings with RoPE (rotary embeddings) and compare validation loss at the same step count.
Train on the BPE tokens from BL-16 instead of characters. Compare bits per character, not loss per token, because the token counts differ.
Tie the input embedding and output head weights and see what it does to parameter count and loss.

Terms in plain words

Self-attention
Each position builds a query, a key and a value; it scores every key against its query and takes a weighted average of the values.
Causal mask
Sets the scores for future positions to minus infinity before the softmax, so a token can only look at itself and what came before.
Multi-head attention
Several small attention operations run side by side on slices of the vector, so different heads can track different patterns.
Positional embedding
A learned vector added to each token for its position, because attention by itself has no idea of order.
Residual stream
The running vector for each position; every block reads it and adds its output back in, instead of replacing it.