← All Cracked ML projects
PL-A04NOT STARTEDADVANCEDMLPYTHONIMPACT ★★★☆☆

Sparse autoencoder on GPT-2, then steer with its features

Train an SAE (sparse autoencoder) as good as the public ones, then test whether SAE features steer the model better than ActAdd's difference vectors.

EST.32 h
REWARD+400 XP
AFTERBloom + 2

ML-ARENA BENCHMARKS

PROOF: FINDING
METRICINDUSTRYMINE
CE loss recovered at matched L0; sentiment-classifier shift vs perplexity increase on 500 prompts
GOAL · CE loss recovered ≥ Bloom's layer-8 SAE at equal L0; SAE-feature steering matches ActAdd's sentiment shift at ≤ equal perplexity cost (or a documented negative result)
Joseph Bloom's open GPT-2-small residual SAEs (24,576 latents, via SAELens) and ActAdd · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Cache about 200M residual-stream activations from GPT-2 small layer 8. Train two kinds of SAE (a wide autoencoder that rewrites each activation as a sparse sum of learned feature directions): ReLU-L1, where sparsity comes from an L1 penalty, and TopK, which keeps only the K largest latents. Use decoder-norm constraints (keep each feature direction at unit length), dead-latent resampling (re-initialise latents that never fire) and geometric-median bias init (start the bias at a robust centre point of the data). Score them on the L0 vs CE-loss-recovered frontier. L0 is how many latents are active per token. CE loss recovered is how much of the model's next-token performance survives when you replace the real activation with the SAE's reconstruction. Then compare two ways of steering (adding a vector to the residual stream to change the model's behaviour): ActAdd (the difference between activations on a pair of prompts) and single SAE features. Measure sentiment shift vs perplexity damage.

Approach

An SAE rewrites one residual-stream vector x∈R768x \in \mathbb{R}^{768} as a sparse combination of learned directions:

f=ReLU⁡(Wenc(x−bdec)+benc),x^=Wdecf+bdecf = \operatorname{ReLU}\big(W_\text{enc}(x - b_\text{dec}) + b_\text{enc}\big), \qquad \hat{x} = W_\text{dec} f + b_\text{dec}

with the ReLU-L1 loss ∥x−x^∥22+λ∑ifi∥Wdec,:,i∥2\lVert x - \hat{x} \rVert_2^2 + \lambda \sum_i f_i \lVert W_{\text{dec},:,i} \rVert_2. Keeping each decoder column at unit norm stops the model from shrinking ff and growing WdecW_\text{dec} to dodge the penalty. After each step, renormalise the columns and, before the step, remove the gradient component parallel to each column. The TopK variant drops the L1 term and keeps only the KK largest pre-activations, so L0 is fixed by construction.

The score that matters is not reconstruction error but what the model does with the reconstruction. Splice x^\hat{x} back into layer 8 and run the rest of GPT-2:

CE recovered=CEzero−CEsaeCEzero−CEclean\text{CE recovered} = \frac{\text{CE}_\text{zero} - \text{CE}_\text{sae}}{\text{CE}_\text{zero} - \text{CE}_\text{clean}}

where zero means the activation is replaced with zeros. Some papers use mean-ablation instead of zero; pick one, state it, and use it for Bloom's SAE too. Compare SAEs only at matched L0, because any SAE recovers more CE when it is allowed more active latents.

Order: activation stream, then Bloom's SAE through your evaluator, then your SAEs, then steering. Do not cache all 200M activations: at 768 fp16 values each, that is about 307 GB. Stream them through a shuffled buffer of a few million instead. Each SAE at 24,576 latents costs on the order of 5×10165 \times 10^{16} FLOPs over 200M tokens, plus the GPT-2 forward passes to make the activations. One 24 GB GPU is enough; budget 10 to 20 GPU-hours for a frontier of 6 to 8 SAEs and the steering sweeps.

VISUALFig. 1
ONE TOKEN'S ACTIVATION, GPT-2 SMALL, LAYER 8 RESID_PRE x768 W_enc(x − b_dec)+ b_enc → 24,576 ReLU + L1or TopK (keep K) f: sparseL0 = active latents W_dec f + b_deccolumns kept at norm 1 = x̂, the reconstruction CE LOSS RECOVERED: SPLICE x̂ BACK IN, RUN THE REST OF THE MODEL recovered part CE_clean CE_sae CE_zero x untouched x replaced by x̂ x replaced by 0 recovered = (CE_zero − CE_sae) / (CE_zero − CE_clean) compare SAEs only at matched L0
Top: the SAE forward pass. Bottom: CE loss recovered measures where the SAE lands between the clean model and a model with layer 8 wiped out. Why it's here: Objectives 2 to 4 all report this one number, and it is easy to compute it with the wrong baseline.

Where people get stuck

  • Different residual stream. Bloom's SAEs were trained on TransformerLens's processed GPT-2, which centres the writing weights by default. Hidden states from Hugging Face GPT-2 differ. Use the same model loading as the SAE's config says.
  • Wrong context length. SAEs are trained at a fixed context length. Evaluating on longer contexts, or including the BOS position with its huge norm, makes everything look worse.
  • Dead latents from a bad start. Initialise bdecb_\text{dec} at the geometric median, initialise WencW_\text{enc} as the transpose of WdecW_\text{dec}, and resample latents that have not fired in a few million tokens.
  • Comparing steering at one coefficient. Each method has its own scale. Sweep coefficients and compare the curves of sentiment shift against perplexity cost, not single points.
  • Classifier hacking. Steered text can fool the sentiment model while turning into repetition. Read 20 samples per setting.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 6
Stuck? Hints stay hidden until you ask.

Objectives

0 / 7 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. CE loss recovered at least as high as Bloom's layer-8 SAE at equal L0, both measured by your code on the same tokens. Record it with `python tools/mlarena.py record PL-A04 "<yours> vs <Bloom> at L0 <n>"`.
  2. SAE-feature steering reaches ActAdd's sentiment shift at no more perplexity cost, or a documented negative result with both trade-off curves.
  3. Fewer than 1% dead latents in your best SAE at the end of training.

Side quests

Train a JumpReLU or gated SAE at the same L0 and add it to the frontier.
Feature splitting: train a 4× wider SAE and track what your sentiment feature splits into.

Terms in plain words

Sparse autoencoder (SAE)
An autoencoder with many more hidden units than inputs, trained so only a few fire at once, which tends to give interpretable features.
L0
The number of non-zero latents per input; lower means sparser.
CE loss recovered
How much of the original model's cross-entropy performance is kept when the SAE reconstruction replaces the real activation.
Dead latent
An SAE unit that never activates, so it wastes capacity and gets re-initialised.
Steering
Adding a chosen vector to a model's activations during generation to push its output in a direction, such as more positive.
ActAdd
A steering method that uses the activation difference between two contrasting prompts, such as 'Love' minus 'Hate'.
Geometric median
The point that minimises the sum of distances to all data points; unlike the mean, a few outliers barely move it.
TopK activation
Keep the K largest pre-activations of each input and set the rest to zero, so L0 is exactly K.

Why it rates ★★★☆☆

SAEs plus a controlled steering comparison is current interpretability work, though it would not be a headline result.