← All Cracked ML projects
PL-W05NOT STARTEDWOWMLPYTHONIMPACT ★★★★★

Tiny Recursive Model on ARC-AGI: reimplement, then improve

A 7M-parameter recursive net scores 45% on ARC-AGI-1. Rebuild it from the paper and push ARC-AGI-2.

EST.75 h
REWARD+600 XP
AFTERJolicoeur-Martineau + 2

ML-ARENA BENCHMARKS

PROOF: LEADERBOARD
METRICINDUSTRYMINE
ARC-AGI-1 and ARC-AGI-2 public-eval accuracy (pass@2: a task counts as solved if either of two allowed guesses is right)
GOAL · ≥ 40% on ARC-AGI-1 with your reimplementation; > 8% on ARC-AGI-2; an ARC Prize Paper Award or a next-year Kaggle entry
Jolicoeur-Martineau 2025 TRM: 45% ARC-AGI-1, 8% ARC-AGI-2 with ~7M params—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Reimplement TRM (Tiny Recursive Model) from the paper, not the repo. It is a 2-layer net applied recursively (the same small network runs many times, refining its answer each round) with deep supervision (a loss on the answer after every round), ACT halting (adaptive computation time: the model learns when to stop refining), heavy dihedral/colour augmentation (rotations, flips and colour swaps of each puzzle) and test-time voting (solve many augmented copies and take the most common answer). Reproduce the ARC-AGI-1 evaluation score. Then test one idea on ARC-AGI-2, for example a better augmentation-consistency vote or a test-time-training schedule (briefly training on each test puzzle's own examples). Compute is flagged: the official recipe takes about 4×H100 for 3 days per run.

Approach

TRM keeps three tensors: the embedded puzzle xx, the current answer yy and a latent zz. One small network ff, 2 layers deep, does every update. A recursion process runs

z←f(x+y+z)    (n=6 times),y←f(y+z),z \leftarrow f(x + y + z) \;\; (n = 6 \text{ times}), \qquad y \leftarrow f(y + z),

and that process repeats T=3T = 3 times. The first T−1T - 1 repeats run without gradients and only improve yy and zz. Backpropagation goes through the last one only. Each supervision step is scored with

L=CE(y^,y⋆)+BCE(q^, 1[y^=y⋆]),\mathcal{L} = \mathrm{CE}(\hat y, y^\star) + \mathrm{BCE}\bigl(\hat q,\ \mathbb{1}[\hat y = y^\star]\bigr),

where q^\hat q is the halting head. Then yy and zz are detached and carried into the next step, up to Nsup=16N_{\text{sup}} = 16. One supervision step reaches an effective depth of T(n+1)×2=42T(n + 1) \times 2 = 42 layers. The paper's ARC numbers are 44.6% on ARC-AGI-1 and 7.8% on ARC-AGI-2 for the 7M-parameter attention version.

One detail decides what "reproduce" means. Like HRM before it, TRM gives each puzzle a learned embedding and trains on the demonstration pairs of the evaluation tasks too, never their test outputs. The ARC Prize team's HRM analysis spells this out. Your public-eval reproduction should follow the same protocol. A Kaggle entry cannot, because the private tasks are unseen, so training on them has to happen inside the notebook.

Compute is the main cost. The paper's ARC run takes about 3 days on 4 H100s, roughly 290 H100-hours per run. Debug on Sudoku-Extreme or a small subset first, where a bug costs hours, not days. The test-time vote is the other cost: the paper solves 1,000 augmented copies of each puzzle. In the HRM analysis, 300 augmentations got close to the maximum score. Measure the same curve for TRM before you plan a Kaggle budget.

OVERVIEWFig. 1
STATE x puzzle y answer z latent REPEAT T = 3 TIMES pass 1 pass 2 no gradient: only improves y and z pass 3, with gradient repeat n = 6: z ← f(x + y + z) then once: y ← f(y + z) f is one 2-layer network, shared by every update. Per step: T(n + 1) × 2 layers = 3 × 7 × 2 = 42 layers of depth. SCORE THIS STEP ŷ = head(y) q̂ = halt(y) CE(ŷ, y*) + BCE(q̂, [ŷ = y*]) then optimiser step detach y and z, carry them into the next supervision step up to N_sup = 16 steps per batch; training moves on early when q̂ > 0 at test time: solve ~1000 augmented copies, undo each augmentation, vote NUMBERS FROM JOLICOEUR-MARTINEAU 2025 (ARXIV 2510.04871)
One supervision step of TRM. The same 2-layer network updates z six times and y once, three times over, with gradients only on the last pass. The step is scored, then y and z carry into the next step. Why it's here: Most reimplementation bugs are in which passes keep gradients and where tensors are detached; this is the map to check your code against.

Where people get stuck

  • Gradients through every pass. Backpropagating through all TT repeats multiplies memory and changes what the model learns. Wrap the first T−1T - 1 in torch.no_grad() and check that the autograd graph only covers the last.
  • Forgetting to detach. If yy and zz keep their graph into the next supervision step, memory grows with every step and gradients flow where the paper says they don't.
  • Augmentations that don't invert. A vote only works if every prediction is mapped back to the original frame. One wrong inverse, often a transposed rotation or a colour map applied the wrong way, quietly splits the vote.
  • Comparing protocols. A score from training on evaluation demonstrations is not comparable with one that never saw them. Say which protocol every number uses.
  • Noise at 8%. ARC-AGI-2's public evaluation set is 120 tasks, so one task is about 0.8 points. Report seeds, and don't claim a gain smaller than the spread between them.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 7
Stuck? Hints stay hidden until you ask.

Objectives

0 / 8 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. At least 40% pass@2 on the ARC-AGI-1 public evaluation set with your own reimplementation, and above 8% on ARC-AGI-2, recorded with `python tools/mlarena.py record PL-W05 "<ARC-AGI-1>% / <ARC-AGI-2>%"`.
  2. The ARC-AGI-2 gain comes from your idea: same seeds, same compute, with an ablation table that switches it off.
  3. Code public under a permissive licence, plus a paper-track submission or a Kaggle entry that runs within the track's limits.

Side quests

Train only on training tasks and add test-time training on each evaluation puzzle's demonstrations. The gap to the standard protocol shows what a Kaggle entry, which never sees private tasks in advance, can expect.
Replace the learned per-puzzle embedding with a small encoder over the demonstration pairs, so the model can work on puzzles it never trained on.

Terms in plain words

ARC-AGI
A benchmark of small coloured-grid puzzles where you must infer a transformation rule from a few examples.
Recursive net
A network that feeds its own output back in and runs again, so a tiny model can do many steps of reasoning.
Deep supervision
Applying the loss after every refinement round, not only at the end.
ACT halting
A learned signal that tells the model when its answer is good enough to stop iterating.
Test-time training
Fine-tuning briefly on the demonstration examples of each test puzzle before answering it.
pass@2
A task is solved if the right answer is among your two submitted attempts.
Puzzle embedding
A learned vector for each puzzle ID that the model receives as input; it only exists for puzzles seen in training.
Weight EMA
A slowly updated running average of the weights, used for evaluation because it is smoother than the raw weights.

Why it rates ★★★★★

Improving a frontier small-model result on ARC-AGI-2 would get a lab's attention immediately.