BEGINNER0/10
0/ 40 SOLVED
INTERMEDIATE0/11
ADVANCED0/13
WOW0/6
0 wonML-ARENA BENCHMARKS
| Status | PROJECT | LEVEL | BENCHMARK | INDUSTRY | MINE | IMPACT |
|---|---|---|---|---|---|---|
| PL-B01.Bandit arena vs the Lai–Robbins bound MLPYTHONafter Lattimore & Szepesvári | BEGINNER | cumulative regret at T=10^5 steps ÷ the Lai–Robbins lower bound, on a 10-arm Bernoulli bandit (each arm pays 1 or 0 with a fixed probability), over 200 seeds; plus regret on a piecewise-stationary suite | The Lai–Robbins bound, as given in Lattimore & Szepesvári, Bandit Algorithms · measured locally by mlarena bench | — | ★☆☆☆☆ | |
| PL-B02.Reproduce double descent with random features MLPYTHONafter Belkin + 1 | BEGINNER | where the test-MSE peak sits and how high it is; best over-parameterised (N > n) test error vs best under-parameterised (N < n) test error | The double-descent curve shape from Belkin et al. 2019 (PNAS) · measured locally | — | ★★☆☆☆ | |
| PL-B03.Softmax regression speed duel on 581k rows MLPYTHONafter UCI Covertype dataset | BEGINNER | the gap between your final training log-loss and sklearn's, and the wall-clock time to reach it (single thread); test ECE | scikit-learn LogisticRegression(solver='lbfgs') · measured locally | — | ★☆☆☆☆ | |
| PL-B04.100 KB language identifier vs fastText MLPYTHONafter fastText language identification + 1 | BEGINNER | macro accuracy (accuracy averaged per language, so each language counts equally) on the WiLI-2018 test paragraphs, for the languages that overlap with lid.176, at a given model size | fastText lid.176.ftz (917 kB, 176 languages) · accuracy measured locally on the same split | — | ★★☆☆☆ | |
| PL-B05.16-colour palette war: k-means & GMM vs Pillow MLPYTHONafter Kodak Lossless True Color Image Suite + 1 | BEGINNER | mean PSNR (peak signal-to-noise ratio in dB; higher means closer to the original) in RGB over the 24 Kodak images at a 16-colour palette; seconds per image | Pillow Image.quantize with MEDIANCUT and with LIBIMAGEQUANT · measured locally | — | ★☆☆☆☆ | |
| PL-B06.Leak hunter: a leakage-audit CLI MLPYTHONafter Kapoor & Narayanan | BEGINNER | planted leaks detected out of 12; false alarms on the 5 clean controls | Kapoor & Narayanan leakage taxonomy (Patterns 2023) · measured locally by mlarena bench | — | ★★☆☆☆ | |
| PL-B07.Byte-exact BPE: match tiktoken, then beat it on code MLPYTHONafter Karpathy + 1 | BEGINNER | token-ID exact-match rate for gpt2 and o200k_base on 50 MB; encoding speed in MB/s; bytes per token on held-out code (more bytes per token means better compression) | tiktoken (claims to be 3–6× faster than comparable open-source tokenisers) · throughput measured locally | — | ★☆☆☆☆ | |
| PL-B08.A 5-gram language model (modified Kneser–Ney) that ties KenLM MLPYTHONafter KenLM Language Model Toolkit + 1 | BEGINNER | word-level perplexity (how surprised the model is on average; lower is better) and bits/char on the held-out text8 slice; peak RAM | KenLM lmplz -o 5 (modified Kneser–Ney) · measured locally on the same split | — | ★☆☆☆☆ | |
| PL-B09.Vector search from scratch: IVF-PQ vs FAISS MLPYTHONafter ANN-Benchmarks + 1FITS NOW | BEGINNER | recall@10 (the fraction of the true 10 nearest neighbours you return) vs queries/sec curve on glove-100-angular; index RAM | FAISS IndexIVFFlat / IndexIVFPQ (ann-benchmarks protocol) · measured locally | — | ★★☆☆☆ | |
| PL-B10.An image codec that is an MLP (NumPy, hand backprop) MLPYTHONafter Dupont et al. + 1 | BEGINNER | PSNR (peak signal-to-noise ratio) at 0.3 bpp (bits per pixel, counting only the size of the weights) on 6 Kodak images; gradient-check relative error | JPEG via Pillow at matched bpp · measured locally (the COIN paper reports beating JPEG at low bitrates) | — | ★★☆☆☆ | |
| PL-I01.Kernel SVM with your own SMO solver, plus random Fourier features, vs LIBSVM MLPYTHONafter Platt + 2 | INTERMEDIATE | a9a test accuracy and training wall-clock at fixed (C, γ), where C is the regularisation strength and γ the RBF kernel width | scikit-learn SVC (which wraps LIBSVM) · measured locally | — | ★★☆☆☆ | |
| PL-I02.t-SNE from scratch with FFT-interpolated gradients MLPYTHONafter openTSNE documentation + 1FITS NOW | INTERMEDIATE | final KL divergence (how badly the 2-D map's neighbour probabilities match the original ones; lower is better); fraction of each point's 10-NN (10 nearest neighbours) preserved; runtime | openTSNE (same perplexity, iterations, init) · measured locally | — | ★★☆☆☆ | |
| PL-I03.Histogram GBDT in NumPy vs LightGBM on 1M HIGGS rows MLPYTHONafter Ke et al. + 1 | INTERMEDIATE | test AUC and training wall-clock at equal hyper-parameters (num_leaves=255, 500 rounds, lr=0.1) | LightGBM (single thread) · measured locally | — | ★★☆☆☆ | |
| PL-I04.Tensor autograd engine, gradient-checked against PyTorch MLPYTHONafter Max Slater + 2 | INTERMEDIATE | max relative gradient error vs torch.autograd (fp64) over 40 ops × random shapes; MLP train-step time vs PyTorch CPU eager mode; whether the depth-50 net trains | PyTorch autograd · measured locally | — | ★★☆☆☆ | |
| PL-I05.word2vec SGNS vs gensim, then a contrastive rematch MLPYTHONafter Mikolov et al. + 1 | INTERMEDIATE | accuracy on questions-words.txt (the Google analogy set, questions like king − man + woman ≈ queen) and training wall-clock, at dim=100, window=5, 5 epochs | gensim Word2Vec (same hyper-parameters) · measured locally | — | ★☆☆☆☆ | |
| PL-I06.Hand-built LSTM vs cuDNN on enwik8 MLPYTHONafter Karpathy + 1 | INTERMEDIATE | test bits/char at equal training steps; training tokens/sec | torch.nn.LSTM (cuDNN) of the same size · measured locally | — | ★★☆☆☆ | |
| PL-I07.GPT-2 in NumPy that matches Hugging Face to 1e-4 MLPYTHONafter Jay Mody | INTERMEDIATE | max |logit diff| vs HF over 50 prompts; decode tokens/s at batch 1 (fp32, CPU); KV-cache speedup at 512 tokens | Hugging Face transformers GPT2LMHeadModel on CPU · measured locally | — | ★★☆☆☆ | |
| PL-I08.Hybrid retrieval that beats BM25 on BEIR MLPYTHONafter Thakur et al. | INTERMEDIATE | nDCG@10 (a ranking score that rewards putting relevant documents near the top of the first 10 results) per dataset and on average; p95 query latency (95% of queries finish faster than this) | BEIR (Thakur et al. 2021) BM25 baseline; Pyserini BM25 · measured locally | — | ★★☆☆☆ | |
| PL-I09.GCN on ogbn-arxiv without PyG MLPYTHONafter OGB node property prediction leaderboard | INTERMEDIATE | test accuracy (mean ± std over 10 seeds) on the official split | OGB leaderboard: GCN 0.7174 ± 0.0029 (OGB team), GraphSAGE 0.7149 | — | ★★☆☆☆ | |
| PL-I10.Logit lens vs tuned lens: reading GPT-2's guesses layer by layer MLPYTHONafter Belrose et al. + 1 | INTERMEDIATE | per-layer KL(final ‖ lens), that is, how far each lens's prediction is from the model's real final prediction, on 1M held-out tokens | trained lenses from the AlignmentResearch tuned-lens package · measured locally | — | ★★☆☆☆ | |
| PL-I11.Grokking, reverse-engineered: the Fourier circuit and beyond MLPYTHONafter Nanda et al. + 1 | INTERMEDIATE | grokking reproduced (test acc > 99% long after train acc reaches 100%); variance explained by the key frequencies; restricted/excluded-loss curves | Nanda et al. 2023 progress measures · reproduced locally | — | ★★★☆☆ | |
| PL-A01.Contamination detective: membership inference on open LLMs MLPYTHONafter Shi et al. + 2 | ADVANCED | AUROC (how well a score separates training members from non-members; 0.5 is chance, 1.0 is perfect) on WikiMIA at text lengths 32/64/128, per method and model; overlap rate per benchmark split | Min-K%++ paper (beats the runner-up by 6.2–10.5 AUROC pts on WikiMIA) · reproduced locally | — | ★★★☆☆ | |
| PL-A02.LoRA + DPO from scratch, tested against TRL MLPYTHONafter Rafailov et al. + 2 | ADVANCED | held-out preference accuracy (how often the implicit reward margin, the model's preference for the chosen over the rejected answer, is > 0); win rate vs SFT under a fixed reward model; peak GPU memory | TRL DPOTrainer + PEFT LoRA (same config) · measured locally | — | ★★☆☆☆ | |
| PL-A03.Othello-GPT: find the board, then edit it MLPYTHONafter Neel Nanda + 1 | ADVANCED | legal-move top-1 rate (how often the model's top predicted move is legal); per-layer linear-probe error; % of interventions whose new legal-move set matches the edited board | Li et al. 2023: linear probe 20.4% error vs non-linear 1.7% (per Nanda's write-up); Nanda's 'mine/theirs' linear probe | — | ★★★☆☆ | |
| PL-A04.Sparse autoencoder on GPT-2, then steer with its features MLPYTHONafter Bloom + 2 | ADVANCED | CE loss recovered at matched L0; sentiment-classifier shift vs perplexity increase on 500 prompts | Joseph Bloom's open GPT-2-small residual SAEs (24,576 latents, via SAELens) and ActAdd · measured locally | — | ★★★☆☆ | |
| PL-A05.Ventilator: reverse-engineer the PID controller MLPYTHONafter Kaggle + 1 | ADVANCED | Kaggle private-leaderboard MAE (mean absolute error), scored on the inspiratory (breathing-in) phase only; % of test breaths reconstructed exactly | Google Brain Ventilator Pressure Prediction private LB (leaderboard); the 1st of 2,650 teams used PID matching + LSTM/CNN-transformer | — | ★★★☆☆ | |
| PL-A06.CIFAR-10 to 94% in seconds: rebuild airbench MLPYTHONafter Keller Jordan + 2 | ADVANCED | mean test accuracy over 25 runs and the wall-clock time to finish on your GPU | airbench94_muon: 94.01% in 2.59 s on an A100 (README); run it on your own GPU to get the reference time | — | ★★★☆☆ | |
| PL-A07.CLIP from scratch: port the weights, then train your own MLPYTHONafter Radford et al. + 1 | ADVANCED | cosine similarity between your embeddings and the reference embeddings; zero-shot top-1 (CIFAR-10, ImageNet-V2); zero-shot accuracy of your from-scratch model | the openai/CLIP package with ViT-B/32 weights · measured locally | — | ★★☆☆☆ | |
| PL-A08.DDPM vs flow matching on a 12-GPU-hour budget MLPYTHONafter Ho et al. + 2 | ADVANCED | FID-50k (Fréchet Inception Distance over 50k generated images; lower means more realistic and varied) vs number of function evaluations (NFE, the number of network calls one sample takes), at a fixed 12 GPU-hour training budget | DDPM paper: FID 3.17 on CIFAR-10 (with a much larger budget); pytorch-fid for scoring | — | ★★★☆☆ | |
| PL-A09.AlphaZero Connect-4, graded by a perfect solver MLPYTHONafter Silver et al. + 1 | ADVANCED | % optimal moves on 1,000 solver-labelled positions; win rate as first player vs the solver; GPU-hours used | Pascal Pons's Connect-4 perfect solver (ground truth) | — | ★★★☆☆ | |
| PL-A10.GPT-2 124M trainer with your own data-parallel + ZeRO-1 MLPYTHONafter Karpathy + 1 | ADVANCED | FineWeb validation loss vs tokens seen; tokens/s scaling efficiency from 1→N GPUs; optimiser memory per GPU | torch DDP (measured locally); llm.c GPT-2 124M reproduction (90 min on 8×A100, ~$20) | — | ★★★☆☆ | |
| PL-A11.Triton kernels on the roofline, up to FlashAttention MLPYTHONafter Triton tutorial + 2 | ADVANCED | attention TFLOP/s at sequence length 4k, head-dim 64, fp16 (forward and backward); LayerNorm's % of peak memory bandwidth | PyTorch scaled_dot_product_attention (SDPA, flash backend) and the Triton fused-attention tutorial · measured locally | — | ★★★☆☆ | |
| PL-A12.A tiny deep-learning compiler: trace, fuse, generate code MLPYTHONafter JAX + 2 | ADVANCED | forward+backward speedup over eager mode on a GPT-2 block and an MLP; kernel count; peak memory | torch.compile (Inductor backend) on the same block · measured locally | — | ★★★☆☆ | |
| PL-A13.LLM serving engine: paged KV, continuous batching, int8 MLPYTHONafter Kwon et al. + 1 | ADVANCED | output tokens/s; p50/p99 time-to-first-token (TTFT, the delay before the first token streams back) under load; int8 perplexity delta on WikiText-2 | vLLM on the same GPU · measured locally (nano-vllm README: 1434 vs vLLM 1362 tok/s on an RTX 4070 Laptop) | — | ★★★☆☆ | |
| PL-W01.Chess-GPT: probe board and skill, then find something new MLPYTHONafter Karvonen + 2 | WOW | board-probe accuracy (best layer); skill-intervention win-rate multiplier (how many times more often the model wins after the intervention); new-latent probe accuracy plus causal effect size | Karvonen 2024: board probe 99.6% (16-layer) / 99.1% (8-layer); the skill vector raises win rate up to ~2.6× | — | ★★★★☆ | |
| PL-W02.Scaling scaling laws: Hex, then a game nobody fitted MLPYTHONafter Andy L. Jones + 1 | WOW | extrapolation error of the 9×9 compute frontier when it is predicted from fits on boards ≤ 7×7; fit quality on the new game | Jones 2021, Scaling Scaling Laws with Board Games (frontiers are exponential in performance and board size) | — | ★★★★☆ | |
| PL-W03.BabyLM Strict-Small: beat the baseline on 10M words MLPYTHONafter BabyLM Challenge + 1 | WOW | babylm-eval aggregate score (Strict-Small) and the per-task breakdown | Official BabyLM 2026 Strict-Small baselines and the Hugging Face leaderboard | — | ★★★★☆ | |
| PL-W04.Vesuvius Challenge: ink detection for a Progress Prize MLPYTHONafter Vesuvius Challenge + 1 | WOW | late-submission score on the Kaggle Vesuvius Ink Detection private LB (leaderboard), using the competition metric; an accepted prize submission | Kaggle Vesuvius Challenge Ink Detection (2023, 1,249 teams); scrollprize.org Progress Prizes ($250–$20,000 monthly, $20k best-of-month) | — | ★★★★★ | |
| PL-W05.Tiny Recursive Model on ARC-AGI: reimplement, then improve MLPYTHONafter Jolicoeur-Martineau + 2 | WOW | ARC-AGI-1 and ARC-AGI-2 public-eval accuracy (pass@2: a task counts as solved if either of two allowed guesses is right) | Jolicoeur-Martineau 2025 TRM: 45% ARC-AGI-1, 8% ARC-AGI-2 with ~7M params | — | ★★★★★ | |
| PL-W06.Set a modded-nanogpt world record MLPYTHONafter KellerJordan/modded-nanogpt + 1 | WOW | wall-clock time (or steps, on Track 3) to reach the target val loss on 8×H100, with p < 0.01 over runs | modded-nanogpt records: Track 1 #92 at 0.665 min (2026-08-30); Track 2 #18 at 17.35 min (2025-12-31) | — | ★★★★★ |
INDUSTRY is the library, paper or record you measure against. MINE is your latest recorded result. IMPACT is a harsh hiring-manager rating of a finished version.