Brief
On GPT-2 124M and Pythia-410M, use your own forward hooks (functions that capture a layer's output during the forward pass) to decode each layer's residual stream through the unembedding. The residual stream is the running vector that every layer reads from and adds to; the unembedding is the final matrix that turns that vector into next-token logits. Applying it to early layers is called the logit lens. Then train per-layer affine translators (one learned linear map plus bias for each layer) that minimise KL divergence to the final output distribution; this is the tuned lens. Write a case study on one failure mode of the logit lens.
VICTORY CONDITIONS
- tuned-lens KL below logit-lens KL at every layer, and within 10% of the reference package's KL (per-layer KL(final ‖ lens), that is, how far each lens's prediction is from the model's real final prediction, on 1M held-out tokens).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Residual stream
- The vector for each token that flows through the transformer, with every attention and MLP block adding its output to it.
- Unembedding
- The last matrix in a language model, which maps the final vector to a score for every token in the vocabulary.
- Logit lens
- Applying the unembedding to an intermediate layer to see what the model would predict if it stopped there.
- Tuned lens
- A logit lens with a small learned linear correction per layer, which reads intermediate layers more faithfully.
- KL divergence
- A measure of how different two probability distributions are; zero means identical.
Why it rates ★★☆☆☆
A faithful reproduction shows you can do interpretability work, but it is not a new finding.