Brief
A GPT is a language model that predicts the next token from all the tokens before it. You built one in BL-15 with an LSTM, which squeezes the past into a single hidden state. A transformer does something different: at every position it looks back at all earlier positions directly and decides which ones matter. That mechanism is attention, and it is the new idea here.
Each position makes three vectors from its input: a query (what am I looking for), a key (what do I contain) and a value (what do I pass on). Stack them into matrices and one head computes
where is the key size and is the causal mask: on and below the diagonal, above it. Dividing by keeps the scores from growing with the vector size, so the softmax doesn't saturate. Several heads run in parallel and their outputs are joined back together. A block is attention plus a small MLP, each wrapped in LayerNorm and a residual connection, both of which you met in BL-12.
You'll write the model in PyTorch from nn.Linear, nn.Embedding and nn.LayerNorm up, without
nn.MultiheadAttention or nn.Transformer. Keep the code tidy and save the checkpoint well. This model is the one you
speed up in BL-22, serve in BL-27, look inside in BL-29 and steer in BL-30.
DATA · a Project Gutenberg book
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your validation loss is within 5% of nanoGPT's on the same data, size and step count. Record both with `python tools/mlarena.py record BL-21 "<yours> vs <nanoGPT>"`.
- Your attention matches scaled_dot_product_attention to 1e-5 and passes the "future tokens change nothing" test.
- A 500-character sample from the final model reads like the book, and the saved checkpoint plus model code load cleanly in a fresh script (BL-22, BL-27, BL-29 and BL-30 all start from it).
Side quests
Terms in plain words
- Self-attention
- Each position builds a query, a key and a value; it scores every key against its query and takes a weighted average of the values.
- Causal mask
- Sets the scores for future positions to minus infinity before the softmax, so a token can only look at itself and what came before.
- Multi-head attention
- Several small attention operations run side by side on slices of the vector, so different heads can track different patterns.
- Positional embedding
- A learned vector added to each token for its position, because attention by itself has no idea of order.
- Residual stream
- The running vector for each position; every block reads it and adds its output back in, instead of replacing it.