← All Cracked ML projects
PL-W01NOT STARTEDWOWMLPYTHONIMPACT ★★★★☆

Chess-GPT: probe board and skill, then find something new

Reproduce Karvonen's board-state and Elo-skill probes, then publish a world-model finding nobody has reported.

EST.45 h
REWARD+600 XP
AFTERKarvonen + 2

ML-ARENA BENCHMARKS

PROOF: FINDING
METRICINDUSTRYMINE
board-probe accuracy (best layer); skill-intervention win-rate multiplier (how many times more often the model wins after the intervention); new-latent probe accuracy plus causal effect size
GOAL · reproduce within 0.5 pt and get a ≥ 2× win-rate lift; one new causally validated latent, published
Karvonen 2024: board probe 99.6% (16-layer) / 99.1% (8-layer); the skill vector raises win rate up to ~2.6×—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Train 8- and 16-layer character-level GPTs on Lichess PGN strings (chess games written as plain-text move lists). Reproduce the board-state linear probes (linear classifiers that read the board position out of the model's activations) and the skill-vector intervention (adding a direction to the activations that makes the model play like a stronger or weaker player). The original part: probe for a new latent, meaning an internal variable such as attacked squares, pins, check, or whether legality is computed from the board representation, and test it causally. Write it up as a paper or LessWrong post with code.

Approach

A linear probe for square ss at layer ℓ\ell is a softmax regression on the residual vector hℓ∈R512h_\ell \in \mathbb{R}^{512}:

p^s=softmax⁡(Wshℓ+bs),Ws∈R13×512.\hat p_s = \operatorname{softmax}(W_s h_\ell + b_s), \qquad W_s \in \mathbb{R}^{13 \times 512}.

The 13 classes are "empty" plus six piece types for each side. Label them as "mine" and "yours" relative to the side to move, not white and black, following Nanda's Othello work. The skill direction is a difference of means over residual vectors from turns 25 to 35:

vℓ=1∣H∣∑g∈Hhˉℓ(g)  −  1∣L∣∑g∈Lhˉℓ(g),hℓ←hℓ+α vℓ,v_\ell = \frac{1}{|H|}\sum_{g \in H} \bar h_\ell(g) \;-\; \frac{1}{|L|}\sum_{g \in L} \bar h_\ell(g), \qquad h_\ell \leftarrow h_\ell + \alpha\, v_\ell,

where HH and LL are the high- and low-skill games. The headline number is the ratio of win rates with and without the edit. Most of that ratio comes from games that start from 20 random moves. There the 16-layer model goes from 16.7% to 43.2%. On normal openings it only moves from 69.6% to 72.3%. Measure both, and don't compare a ratio from one setting with the other.

Build it in this order: data, then probes on the released checkpoints, then your own training run, then the interventions. Probing first gives you a working read-out pipeline in a day. The training run is the expensive part. At C≈6NDC \approx 6ND with D=600,000×100×1,023≈6.1×1010D = 600{,}000 \times 100 \times 1{,}023 \approx 6.1 \times 10^{10} characters, the 8-layer model costs about 9×10189 \times 10^{18} FLOPs and the 16-layer about 1.8×10191.8 \times 10^{19}. That is roughly 50 and 100 GPU-hours at a sustained 50 TFLOP/s. Probe training is cheap: the repo's author reports about 10 minutes on an RTX 3050 for the 8-layer model. Games against Stockfish are CPU-bound, so run them in parallel processes.

For the new latent, the probe is only half the result. High probe accuracy shows the information can be read out. It does not show the model uses it. The finding is the edit: change the latent and the moves change the way the latent predicts.

OVERVIEWFig. 1
INPUT ;1.e4 e5 2 . Nf3 … probes read at each "." GPT, 8 or 16 LAYERS layer L: residual vector h (512-d) → next character 1 · READ (correlation) Linear probe per square s: softmax(W_s h + b_s) W_s is 13 × 512: empty, or one of 6 pieces for "mine" / "yours" → 8 × 8 board 2 · EDIT Add a direction h′ = h + α · d̂ (α can be negative) d̂: a probe row (board edit) or mean(high Elo) − mean(low Elo) at layers L…, then run on 3 · CHECK (causation) Did behaviour move? legal-move rate on edited boards win rate vs Stockfish level 0 same games, with and without the edit, plus a random direction A probe that reads a latent shows the information is there. Only an edit that changes moves in the predicted way shows the model uses it.
Read, edit, check. Probes read the residual vector at each '.'. Edits add a direction there and let the forward pass continue. The check compares behaviour with and without the edit. Why it's here: The new-latent result needs all three boxes. Most failed attempts stop after the first.

Where people get stuck

  • Off-by-one positions. The '.' comes before White's move, so the board you label is the one after Black's last move. Check a dozen positions by hand against python-chess before training any probe.
  • Accuracy that comes for free. Most squares are empty or unchanged from the start, so a random-init model already scores about 75%. Report per-class F1 and the control, not only overall accuracy.
  • Win-rate noise. At a 50% win rate the standard error over nn games is 0.25/n\sqrt{0.25/n}, about 1.6 points at 1,000 games. Pair the arms by using the same openings and Stockfish settings. Changing the time or node limit moves the baseline more than many edits do.
  • Edit strength. The paper's positive skill edit uses a small scale (0.1) and the board edit a large one (1.5 to 2). Too large an edit breaks the model, and the legal-move rate on the original board falls. Always report that number next to the effect you want.
  • Claiming novelty too early. Pins, threats and legal moves are already defined in the repo. Search the paper's citations and follow-ups before you pick your latent.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 7
Stuck? Hints stay hidden until you ask.

Objectives

0 / 8 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Board probes within 0.5 points of 99.1% (8-layer) and 99.6% (16-layer) at the best layer, and a skill vector that at least doubles the win rate on random-opening games against Stockfish level 0, recorded with `python tools/mlarena.py record PL-W01 "<probe 8L>/<probe 16L>, <lift>×"`.
  2. One new latent whose probe beats both controls and whose edit changes behaviour more than a random direction of equal norm, with a 95% interval that excludes zero.
  3. A public write-up with code and probe weights that reproduces every number.

Side quests

Use the training checkpoints to find when your latent becomes linearly readable, and whether that happens before or after the legal-move rate passes 99%.
Train a sparse autoencoder on the layer where your probe peaks and check whether any single feature tracks your latent better than the linear probe.

Terms in plain words

PGN
Portable Game Notation, the standard text format for chess games, such as '1.e4 e5 2.Nf3'.
Linear probe
A linear classifier trained on a layer's activations to test whether some information, like the board, can be read out simply.
Skill vector
A direction in activation space that, when added, makes the model play like a higher-rated player.
Latent
A hidden variable the model computes internally but never outputs directly.
Causal validation
Showing a latent matters by editing it and seeing the model's behaviour change, not just by reading it with a probe.
Elo
A chess rating system where a higher number means a stronger player.
Contrastive vector
The average activation on one group of inputs minus the average on another, used as a direction to steer the model.
Control task
A probe trained on a model or labels that should carry no signal, used to show how much accuracy a probe gets for free.

Why it rates ★★★★☆

A reproduction plus a new, causally validated world-model result is publishable MI (mechanistic interpretability) work.