← All Base Form ML projects
BL-15NOT STARTEDBASE FORM · DEEP LEARNINGMLPYTHON

Style writer

A character-level RNN/LSTM that writes text in the style of a book you love.

EST.8 h
REWARD+100 XP
AFTERAndrej Karpathy

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
gradients; validation loss
GOAL · gradients within 1e-5; beat the n-gram
PyTorch LSTM · n-gram baseline—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

A language model predicts the next token from the ones before it. Here a token is a single character. Feed it a book and it picks up spelling, punctuation and a little of the author's voice. Sample from it one character at a time and it writes new text in that style.

Your baseline is an n-gram model. Count how often each character follows each run of n−1n - 1 characters and turn the counts into probabilities. It's fast and hard to beat for small nn, but it remembers exactly n−1n - 1 characters and nothing more. A recurrent neural network (RNN) carries a hidden state forward instead: ht=tanh⁡(Wxhxt+Whhht−1+b)h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b). In principle it can remember much further back.

Training an RNN means unrolling it over a window of characters and running backpropagation through the unrolled graph. This is called backpropagation through time (BPTT). The gradient passes through WhhW_{hh} once per step, so over long windows it tends to vanish or explode. The LSTM adds a separate cell state and gates that decide what to keep, which lets useful gradients survive much longer. Gradient clipping caps the explosions.

Pick any book from Project Gutenberg (gutenberg.org) and download its "Plain Text UTF-8" version. Something long, like a novel of at least 500,000 characters, gives the network enough to learn from. Measure both models by cross-entropy on the last 10% of the book.

VISUALFig. 1
ONE TRAINING WINDOW h from last window no gradient h₁ x₁ y₁ h₂ x₂ y₂ h₃ x₃ y₃ h₄ x₄ y₄ W_hh ← BPTT: gradients flow back to next window
An RNN unrolled over one window. The hidden state flows forward; BPTT sends gradients back along the same chain. The state carried in from the previous window keeps its value but gets no gradient. Why it's here: Objective 5's truncated BPTT is easy to get wrong. This shows where the gradient stops.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.

DATA · a Project Gutenberg book

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. LSTM gradients match PyTorch (or finite differences) within 1e-5, and validation loss beats the best n-gram, recorded with `python tools/mlarena.py record BL-15 "<max grad error>; <val loss> vs <n-gram loss>"`.
  2. Both models are scored in nats per character on the same last 10% of the book.
  3. The README shows samples at three temperatures from the same seed text.

Side quests

Swap the LSTM for a GRU and compare validation loss and speed.
Find a single hidden unit that tracks something readable, like being inside quotes, and plot its value over a passage.

Terms in plain words

Language model
A model that gives a probability for the next token given the tokens before it. Sampling from it one token at a time generates text.
n-gram model
A language model that predicts the next character from only the previous n − 1 characters, using counts from the training text.
Backpropagation through time (BPTT)
Unrolling an RNN over a window of steps and running ordinary backpropagation through the unrolled copy.
LSTM
An RNN cell with a separate memory and three gates that decide what to forget, what to write and what to output, which helps gradients last longer.
Temperature
A number you divide the scores by before softmax when sampling. Low values play safe; high values take more risks.