Brief
Cache about 200M residual-stream activations from GPT-2 small layer 8. Train two kinds of SAE (a wide autoencoder that rewrites each activation as a sparse sum of learned feature directions): ReLU-L1, where sparsity comes from an L1 penalty, and TopK, which keeps only the K largest latents. Use decoder-norm constraints (keep each feature direction at unit length), dead-latent resampling (re-initialise latents that never fire) and geometric-median bias init (start the bias at a robust centre point of the data). Score them on the L0 vs CE-loss-recovered frontier. L0 is how many latents are active per token. CE loss recovered is how much of the model's next-token performance survives when you replace the real activation with the SAE's reconstruction. Then compare two ways of steering (adding a vector to the residual stream to change the model's behaviour): ActAdd (the difference between activations on a pair of prompts) and single SAE features. Measure sentiment shift vs perplexity damage.
Approach
An SAE rewrites one residual-stream vector as a sparse combination of learned directions:
with the ReLU-L1 loss . Keeping each decoder column at unit norm stops the model from shrinking and growing to dodge the penalty. After each step, renormalise the columns and, before the step, remove the gradient component parallel to each column. The TopK variant drops the L1 term and keeps only the largest pre-activations, so L0 is fixed by construction.
The score that matters is not reconstruction error but what the model does with the reconstruction. Splice back into layer 8 and run the rest of GPT-2:
where zero means the activation is replaced with zeros. Some papers use mean-ablation instead of zero; pick one, state it, and use it for Bloom's SAE too. Compare SAEs only at matched L0, because any SAE recovers more CE when it is allowed more active latents.
Order: activation stream, then Bloom's SAE through your evaluator, then your SAEs, then steering. Do not cache all 200M activations: at 768 fp16 values each, that is about 307 GB. Stream them through a shuffled buffer of a few million instead. Each SAE at 24,576 latents costs on the order of FLOPs over 200M tokens, plus the GPT-2 forward passes to make the activations. One 24 GB GPU is enough; budget 10 to 20 GPU-hours for a frontier of 6 to 8 SAEs and the steering sweeps.
Where people get stuck
- Different residual stream. Bloom's SAEs were trained on TransformerLens's processed GPT-2, which centres the writing weights by default. Hidden states from Hugging Face GPT-2 differ. Use the same model loading as the SAE's config says.
- Wrong context length. SAEs are trained at a fixed context length. Evaluating on longer contexts, or including the BOS position with its huge norm, makes everything look worse.
- Dead latents from a bad start. Initialise at the geometric median, initialise as the transpose of , and resample latents that have not fired in a few million tokens.
- Comparing steering at one coefficient. Each method has its own scale. Sweep coefficients and compare the curves of sentiment shift against perplexity cost, not single points.
- Classifier hacking. Steered text can fool the sentiment model while turning into repetition. Read 20 samples per setting.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- CE loss recovered at least as high as Bloom's layer-8 SAE at equal L0, both measured by your code on the same tokens. Record it with `python tools/mlarena.py record PL-A04 "<yours> vs <Bloom> at L0 <n>"`.
- SAE-feature steering reaches ActAdd's sentiment shift at no more perplexity cost, or a documented negative result with both trade-off curves.
- Fewer than 1% dead latents in your best SAE at the end of training.
Side quests
Terms in plain words
- Sparse autoencoder (SAE)
- An autoencoder with many more hidden units than inputs, trained so only a few fire at once, which tends to give interpretable features.
- L0
- The number of non-zero latents per input; lower means sparser.
- CE loss recovered
- How much of the original model's cross-entropy performance is kept when the SAE reconstruction replaces the real activation.
- Dead latent
- An SAE unit that never activates, so it wastes capacity and gets re-initialised.
- Steering
- Adding a chosen vector to a model's activations during generation to push its output in a direction, such as more positive.
- ActAdd
- A steering method that uses the activation difference between two contrasting prompts, such as 'Love' minus 'Hate'.
- Geometric median
- The point that minimises the sum of distances to all data points; unlike the mean, a few outliers barely move it.
- TopK activation
- Keep the K largest pre-activations of each input and set the rest to zero, so L0 is exactly K.
Why it rates ★★★☆☆
SAEs plus a controlled steering comparison is current interpretability work, though it would not be a headline result.