← All Cracked ML projects
PL-I10NOT STARTEDINTERMEDIATEMLPYTHONIMPACT ★★☆☆☆

Logit lens vs tuned lens: reading GPT-2's guesses layer by layer

Also known as: Logit lens vs tuned lens, layer by layer

Read what GPT-2 is 'thinking' at every layer, then train probes that read it better.

EST.14 h
REWARD+300 XP
AFTERBelrose et al. + 1

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
per-layer KL(final ‖ lens), that is, how far each lens's prediction is from the model's real final prediction, on 1M held-out tokens
GOAL · tuned-lens KL below logit-lens KL at every layer, and within 10% of the reference package's KL
trained lenses from the AlignmentResearch tuned-lens package · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

On GPT-2 124M and Pythia-410M, use your own forward hooks (functions that capture a layer's output during the forward pass) to decode each layer's residual stream through the unembedding. The residual stream is the running vector that every layer reads from and adds to; the unembedding is the final matrix that turns that vector into next-token logits. Applying it to early layers is called the logit lens. Then train per-layer affine translators (one learned linear map plus bias for each layer) that minimise KL divergence to the final output distribution; this is the tuned lens. Write a case study on one failure mode of the logit lens.

VICTORY CONDITIONS

  1. tuned-lens KL below logit-lens KL at every layer, and within 10% of the reference package's KL (per-layer KL(final ‖ lens), that is, how far each lens's prediction is from the model's real final prediction, on 1M held-out tokens).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Residual stream
The vector for each token that flows through the transformer, with every attention and MLP block adding its output to it.
Unembedding
The last matrix in a language model, which maps the final vector to a score for every token in the vocabulary.
Logit lens
Applying the unembedding to an intermediate layer to see what the model would predict if it stopped there.
Tuned lens
A logit lens with a small learned linear correction per layer, which reads intermediate layers more faithfully.
KL divergence
A measure of how different two probability distributions are; zero means identical.

Why it rates ★★☆☆☆

A faithful reproduction shows you can do interpretability work, but it is not a new finding.