Brief
Train 8- and 16-layer character-level GPTs on Lichess PGN strings (chess games written as plain-text move lists). Reproduce the board-state linear probes (linear classifiers that read the board position out of the model's activations) and the skill-vector intervention (adding a direction to the activations that makes the model play like a stronger or weaker player). The original part: probe for a new latent, meaning an internal variable such as attacked squares, pins, check, or whether legality is computed from the board representation, and test it causally. Write it up as a paper or LessWrong post with code.
Approach
A linear probe for square at layer is a softmax regression on the residual vector :
The 13 classes are "empty" plus six piece types for each side. Label them as "mine" and "yours" relative to the side to move, not white and black, following Nanda's Othello work. The skill direction is a difference of means over residual vectors from turns 25 to 35:
where and are the high- and low-skill games. The headline number is the ratio of win rates with and without the edit. Most of that ratio comes from games that start from 20 random moves. There the 16-layer model goes from 16.7% to 43.2%. On normal openings it only moves from 69.6% to 72.3%. Measure both, and don't compare a ratio from one setting with the other.
Build it in this order: data, then probes on the released checkpoints, then your own training run, then the interventions. Probing first gives you a working read-out pipeline in a day. The training run is the expensive part. At with characters, the 8-layer model costs about FLOPs and the 16-layer about . That is roughly 50 and 100 GPU-hours at a sustained 50 TFLOP/s. Probe training is cheap: the repo's author reports about 10 minutes on an RTX 3050 for the 8-layer model. Games against Stockfish are CPU-bound, so run them in parallel processes.
For the new latent, the probe is only half the result. High probe accuracy shows the information can be read out. It does not show the model uses it. The finding is the edit: change the latent and the moves change the way the latent predicts.
Where people get stuck
- Off-by-one positions. The '.' comes before White's move, so the board you label is the one after Black's last move. Check a dozen positions by hand against python-chess before training any probe.
- Accuracy that comes for free. Most squares are empty or unchanged from the start, so a random-init model already scores about 75%. Report per-class F1 and the control, not only overall accuracy.
- Win-rate noise. At a 50% win rate the standard error over games is , about 1.6 points at 1,000 games. Pair the arms by using the same openings and Stockfish settings. Changing the time or node limit moves the baseline more than many edits do.
- Edit strength. The paper's positive skill edit uses a small scale (0.1) and the board edit a large one (1.5 to 2). Too large an edit breaks the model, and the legal-move rate on the original board falls. Always report that number next to the effect you want.
- Claiming novelty too early. Pins, threats and legal moves are already defined in the repo. Search the paper's citations and follow-ups before you pick your latent.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Board probes within 0.5 points of 99.1% (8-layer) and 99.6% (16-layer) at the best layer, and a skill vector that at least doubles the win rate on random-opening games against Stockfish level 0, recorded with `python tools/mlarena.py record PL-W01 "<probe 8L>/<probe 16L>, <lift>×"`.
- One new latent whose probe beats both controls and whose edit changes behaviour more than a random direction of equal norm, with a 95% interval that excludes zero.
- A public write-up with code and probe weights that reproduces every number.
Side quests
Terms in plain words
- PGN
- Portable Game Notation, the standard text format for chess games, such as '1.e4 e5 2.Nf3'.
- Linear probe
- A linear classifier trained on a layer's activations to test whether some information, like the board, can be read out simply.
- Skill vector
- A direction in activation space that, when added, makes the model play like a higher-rated player.
- Latent
- A hidden variable the model computes internally but never outputs directly.
- Causal validation
- Showing a latent matters by editing it and seeing the model's behaviour change, not just by reading it with a probe.
- Elo
- A chess rating system where a higher number means a stronger player.
- Contrastive vector
- The average activation on one group of inputs minus the average on another, used as a direction to steer the model.
- Control task
- A probe trained on a model or labels that should carry no signal, used to show how much accuracy a probe gets for free.
Why it rates ★★★★☆
A reproduction plus a new, causally validated world-model result is publishable MI (mechanistic interpretability) work.