← All Cracked ML projects
PL-I06NOT STARTEDINTERMEDIATEMLPYTHONIMPACT ★★☆☆☆

Hand-built LSTM vs cuDNN on enwik8

Write the LSTM gates yourself, keep up with cuDNN's speed, and beat your own Kneser–Ney model's bits per character.

EST.12 h
REWARD+300 XP
AFTERKarpathy + 1

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
test bits/char at equal training steps; training tokens/sec
GOAL · bpc (bits per character) within 0.02 of nn.LSTM at ≥ 50% of its throughput; beat PL-B08's Kneser–Ney (KN) bpc
torch.nn.LSTM (cuDNN) of the same size · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In PyTorch tensors without nn.LSTM, write an LSTM cell that computes all four gates with one fused matmul. Add stacked layers, truncated BPTT with carried state (backpropagate through a fixed number of time steps, but carry the hidden state forward from one chunk to the next), gradient clipping and weight tying (share weights between the input embedding and the output projection). Optionally torch.compile the cell. Train a character-level model on the standard enwik8 90/5/5 split, compare it against nn.LSTM of identical size, and plot it on the PL-B08 ladder.

VICTORY CONDITIONS

  1. bpc (bits per character) within 0.02 of nn.LSTM at ≥ 50% of its throughput; beat PL-B08's Kneser–Ney (KN) bpc (test bits/char at equal training steps; training tokens/sec).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

cuDNN
NVIDIA's library of hand-optimised GPU kernels that nn.LSTM uses under the hood.
Fused gate matmul
Computing the input, forget, cell and output gates with one big matrix multiply instead of four small ones.
Truncated BPTT
Backpropagation through time that stops after a fixed number of steps, while the hidden state still flows forward.
Weight tying
Using the same matrix for the input embedding and the output layer, which saves parameters and often helps.
enwik8
The first 100 million bytes of an English Wikipedia dump, a standard character-level benchmark.

Why it rates ★★☆☆☆

RNN internals are now niche, but matching cuDNN shows you understand fused kernels and BPTT.