Brief
In PyTorch tensors without nn.LSTM, write an LSTM cell that computes all four gates with one fused matmul. Add stacked layers, truncated BPTT with carried state (backpropagate through a fixed number of time steps, but carry the hidden state forward from one chunk to the next), gradient clipping and weight tying (share weights between the input embedding and the output projection). Optionally torch.compile the cell. Train a character-level model on the standard enwik8 90/5/5 split, compare it against nn.LSTM of identical size, and plot it on the PL-B08 ladder.
VICTORY CONDITIONS
- bpc (bits per character) within 0.02 of nn.LSTM at ≥ 50% of its throughput; beat PL-B08's Kneser–Ney (KN) bpc (test bits/char at equal training steps; training tokens/sec).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- cuDNN
- NVIDIA's library of hand-optimised GPU kernels that nn.LSTM uses under the hood.
- Fused gate matmul
- Computing the input, forget, cell and output gates with one big matrix multiply instead of four small ones.
- Truncated BPTT
- Backpropagation through time that stops after a fixed number of steps, while the hidden state still flows forward.
- Weight tying
- Using the same matrix for the input embedding and the output layer, which saves parameters and often helps.
- enwik8
- The first 100 million bytes of an English Wikipedia dump, a standard character-level benchmark.
Why it rates ★★☆☆☆
RNN internals are now niche, but matching cuDNN shows you understand fused kernels and BPTT.