Brief
Reimplement TRM (Tiny Recursive Model) from the paper, not the repo. It is a 2-layer net applied recursively (the same small network runs many times, refining its answer each round) with deep supervision (a loss on the answer after every round), ACT halting (adaptive computation time: the model learns when to stop refining), heavy dihedral/colour augmentation (rotations, flips and colour swaps of each puzzle) and test-time voting (solve many augmented copies and take the most common answer). Reproduce the ARC-AGI-1 evaluation score. Then test one idea on ARC-AGI-2, for example a better augmentation-consistency vote or a test-time-training schedule (briefly training on each test puzzle's own examples). Compute is flagged: the official recipe takes about 4×H100 for 3 days per run.
Approach
TRM keeps three tensors: the embedded puzzle , the current answer and a latent . One small network , 2 layers deep, does every update. A recursion process runs
and that process repeats times. The first repeats run without gradients and only improve and . Backpropagation goes through the last one only. Each supervision step is scored with
where is the halting head. Then and are detached and carried into the next step, up to . One supervision step reaches an effective depth of layers. The paper's ARC numbers are 44.6% on ARC-AGI-1 and 7.8% on ARC-AGI-2 for the 7M-parameter attention version.
One detail decides what "reproduce" means. Like HRM before it, TRM gives each puzzle a learned embedding and trains on the demonstration pairs of the evaluation tasks too, never their test outputs. The ARC Prize team's HRM analysis spells this out. Your public-eval reproduction should follow the same protocol. A Kaggle entry cannot, because the private tasks are unseen, so training on them has to happen inside the notebook.
Compute is the main cost. The paper's ARC run takes about 3 days on 4 H100s, roughly 290 H100-hours per run. Debug on Sudoku-Extreme or a small subset first, where a bug costs hours, not days. The test-time vote is the other cost: the paper solves 1,000 augmented copies of each puzzle. In the HRM analysis, 300 augmentations got close to the maximum score. Measure the same curve for TRM before you plan a Kaggle budget.
Where people get stuck
- Gradients through every pass. Backpropagating through all repeats multiplies memory and changes what the
model learns. Wrap the first in
torch.no_grad()and check that the autograd graph only covers the last. - Forgetting to detach. If and keep their graph into the next supervision step, memory grows with every step and gradients flow where the paper says they don't.
- Augmentations that don't invert. A vote only works if every prediction is mapped back to the original frame. One wrong inverse, often a transposed rotation or a colour map applied the wrong way, quietly splits the vote.
- Comparing protocols. A score from training on evaluation demonstrations is not comparable with one that never saw them. Say which protocol every number uses.
- Noise at 8%. ARC-AGI-2's public evaluation set is 120 tasks, so one task is about 0.8 points. Report seeds, and don't claim a gain smaller than the spread between them.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- At least 40% pass@2 on the ARC-AGI-1 public evaluation set with your own reimplementation, and above 8% on ARC-AGI-2, recorded with `python tools/mlarena.py record PL-W05 "<ARC-AGI-1>% / <ARC-AGI-2>%"`.
- The ARC-AGI-2 gain comes from your idea: same seeds, same compute, with an ablation table that switches it off.
- Code public under a permissive licence, plus a paper-track submission or a Kaggle entry that runs within the track's limits.
Side quests
Terms in plain words
- ARC-AGI
- A benchmark of small coloured-grid puzzles where you must infer a transformation rule from a few examples.
- Recursive net
- A network that feeds its own output back in and runs again, so a tiny model can do many steps of reasoning.
- Deep supervision
- Applying the loss after every refinement round, not only at the end.
- ACT halting
- A learned signal that tells the model when its answer is good enough to stop iterating.
- Test-time training
- Fine-tuning briefly on the demonstration examples of each test puzzle before answering it.
- pass@2
- A task is solved if the right answer is among your two submitted attempts.
- Puzzle embedding
- A learned vector for each puzzle ID that the model receives as input; it only exists for puzzles seen in training.
- Weight EMA
- A slowly updated running average of the weights, used for evaluation because it is smoother than the raw weights.
Why it rates ★★★★★
Improving a frontier small-model result on ARC-AGI-2 would get a lab's attention immediately.