Brief
A language model predicts the next token from the ones before it. Here a token is a single character. Feed it a book and it picks up spelling, punctuation and a little of the author's voice. Sample from it one character at a time and it writes new text in that style.
Your baseline is an n-gram model. Count how often each character follows each run of characters and turn the counts into probabilities. It's fast and hard to beat for small , but it remembers exactly characters and nothing more. A recurrent neural network (RNN) carries a hidden state forward instead: . In principle it can remember much further back.
Training an RNN means unrolling it over a window of characters and running backpropagation through the unrolled graph. This is called backpropagation through time (BPTT). The gradient passes through once per step, so over long windows it tends to vanish or explode. The LSTM adds a separate cell state and gates that decide what to keep, which lets useful gradients survive much longer. Gradient clipping caps the explosions.
Pick any book from Project Gutenberg (gutenberg.org) and download its "Plain Text UTF-8" version. Something long, like a novel of at least 500,000 characters, gives the network enough to learn from. Measure both models by cross-entropy on the last 10% of the book.
DATA · a Project Gutenberg book
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- LSTM gradients match PyTorch (or finite differences) within 1e-5, and validation loss beats the best n-gram, recorded with `python tools/mlarena.py record BL-15 "<max grad error>; <val loss> vs <n-gram loss>"`.
- Both models are scored in nats per character on the same last 10% of the book.
- The README shows samples at three temperatures from the same seed text.
Side quests
Terms in plain words
- Language model
- A model that gives a probability for the next token given the tokens before it. Sampling from it one token at a time generates text.
- n-gram model
- A language model that predicts the next character from only the previous n − 1 characters, using counts from the training text.
- Backpropagation through time (BPTT)
- Unrolling an RNN over a window of steps and running ordinary backpropagation through the unrolled copy.
- LSTM
- An RNN cell with a separate memory and three gates that decide what to forget, what to write and what to output, which helps gradients last longer.
- Temperature
- A number you divide the scores by before softmax when sampling. Low values play safe; high values take more risks.