0/ 40 SOLVED
BEGINNER0/10
INTERMEDIATE0/11
ADVANCED0/13
WOW0/6
0 wonML-ARENA BENCHMARKS
StatusPROJECTLEVELBENCHMARKINDUSTRYMINEIMPACT
PL-B01.Bandit arena vs the Lai–Robbins bound
MLPYTHONafter Lattimore & Szepesvári
BEGINNERcumulative regret at T=10^5 steps ÷ the Lai–Robbins lower bound, on a 10-arm Bernoulli bandit (each arm pays 1 or 0 with a fixed probability), over 200 seeds; plus regret on a piecewise-stationary suiteThe Lai–Robbins bound, as given in Lattimore & Szepesvári, Bandit Algorithms · measured locally by mlarena bench—★☆☆☆☆
PL-B02.Reproduce double descent with random features
MLPYTHONafter Belkin + 1
BEGINNERwhere the test-MSE peak sits and how high it is; best over-parameterised (N > n) test error vs best under-parameterised (N < n) test errorThe double-descent curve shape from Belkin et al. 2019 (PNAS) · measured locally—★★☆☆☆
PL-B03.Softmax regression speed duel on 581k rows
MLPYTHONafter UCI Covertype dataset
BEGINNERthe gap between your final training log-loss and sklearn's, and the wall-clock time to reach it (single thread); test ECEscikit-learn LogisticRegression(solver='lbfgs') · measured locally—★☆☆☆☆
PL-B04.100 KB language identifier vs fastText
MLPYTHONafter fastText language identification + 1
BEGINNERmacro accuracy (accuracy averaged per language, so each language counts equally) on the WiLI-2018 test paragraphs, for the languages that overlap with lid.176, at a given model sizefastText lid.176.ftz (917 kB, 176 languages) · accuracy measured locally on the same split—★★☆☆☆
PL-B05.16-colour palette war: k-means & GMM vs Pillow
MLPYTHONafter Kodak Lossless True Color Image Suite + 1
BEGINNERmean PSNR (peak signal-to-noise ratio in dB; higher means closer to the original) in RGB over the 24 Kodak images at a 16-colour palette; seconds per imagePillow Image.quantize with MEDIANCUT and with LIBIMAGEQUANT · measured locally—★☆☆☆☆
PL-B06.Leak hunter: a leakage-audit CLI
MLPYTHONafter Kapoor & Narayanan
BEGINNERplanted leaks detected out of 12; false alarms on the 5 clean controlsKapoor & Narayanan leakage taxonomy (Patterns 2023) · measured locally by mlarena bench—★★☆☆☆
PL-B07.Byte-exact BPE: match tiktoken, then beat it on code
MLPYTHONafter Karpathy + 1
BEGINNERtoken-ID exact-match rate for gpt2 and o200k_base on 50 MB; encoding speed in MB/s; bytes per token on held-out code (more bytes per token means better compression)tiktoken (claims to be 3–6× faster than comparable open-source tokenisers) · throughput measured locally—★☆☆☆☆
PL-B08.A 5-gram language model (modified Kneser–Ney) that ties KenLM
MLPYTHONafter KenLM Language Model Toolkit + 1
BEGINNERword-level perplexity (how surprised the model is on average; lower is better) and bits/char on the held-out text8 slice; peak RAMKenLM lmplz -o 5 (modified Kneser–Ney) · measured locally on the same split—★☆☆☆☆
PL-B09.Vector search from scratch: IVF-PQ vs FAISS
MLPYTHONafter ANN-Benchmarks + 1FITS NOW
BEGINNERrecall@10 (the fraction of the true 10 nearest neighbours you return) vs queries/sec curve on glove-100-angular; index RAMFAISS IndexIVFFlat / IndexIVFPQ (ann-benchmarks protocol) · measured locally—★★☆☆☆
PL-B10.An image codec that is an MLP (NumPy, hand backprop)
MLPYTHONafter Dupont et al. + 1
BEGINNERPSNR (peak signal-to-noise ratio) at 0.3 bpp (bits per pixel, counting only the size of the weights) on 6 Kodak images; gradient-check relative errorJPEG via Pillow at matched bpp · measured locally (the COIN paper reports beating JPEG at low bitrates)—★★☆☆☆
PL-I01.Kernel SVM with your own SMO solver, plus random Fourier features, vs LIBSVM
MLPYTHONafter Platt + 2
INTERMEDIATEa9a test accuracy and training wall-clock at fixed (C, γ), where C is the regularisation strength and γ the RBF kernel widthscikit-learn SVC (which wraps LIBSVM) · measured locally—★★☆☆☆
PL-I02.t-SNE from scratch with FFT-interpolated gradients
MLPYTHONafter openTSNE documentation + 1FITS NOW
INTERMEDIATEfinal KL divergence (how badly the 2-D map's neighbour probabilities match the original ones; lower is better); fraction of each point's 10-NN (10 nearest neighbours) preserved; runtimeopenTSNE (same perplexity, iterations, init) · measured locally—★★☆☆☆
PL-I03.Histogram GBDT in NumPy vs LightGBM on 1M HIGGS rows
MLPYTHONafter Ke et al. + 1
INTERMEDIATEtest AUC and training wall-clock at equal hyper-parameters (num_leaves=255, 500 rounds, lr=0.1)LightGBM (single thread) · measured locally—★★☆☆☆
PL-I04.Tensor autograd engine, gradient-checked against PyTorch
MLPYTHONafter Max Slater + 2
INTERMEDIATEmax relative gradient error vs torch.autograd (fp64) over 40 ops × random shapes; MLP train-step time vs PyTorch CPU eager mode; whether the depth-50 net trainsPyTorch autograd · measured locally—★★☆☆☆
PL-I05.word2vec SGNS vs gensim, then a contrastive rematch
MLPYTHONafter Mikolov et al. + 1
INTERMEDIATEaccuracy on questions-words.txt (the Google analogy set, questions like king − man + woman ≈ queen) and training wall-clock, at dim=100, window=5, 5 epochsgensim Word2Vec (same hyper-parameters) · measured locally—★☆☆☆☆
PL-I06.Hand-built LSTM vs cuDNN on enwik8
MLPYTHONafter Karpathy + 1
INTERMEDIATEtest bits/char at equal training steps; training tokens/sectorch.nn.LSTM (cuDNN) of the same size · measured locally—★★☆☆☆
PL-I07.GPT-2 in NumPy that matches Hugging Face to 1e-4
MLPYTHONafter Jay Mody
INTERMEDIATEmax |logit diff| vs HF over 50 prompts; decode tokens/s at batch 1 (fp32, CPU); KV-cache speedup at 512 tokensHugging Face transformers GPT2LMHeadModel on CPU · measured locally—★★☆☆☆
PL-I08.Hybrid retrieval that beats BM25 on BEIR
MLPYTHONafter Thakur et al.
INTERMEDIATEnDCG@10 (a ranking score that rewards putting relevant documents near the top of the first 10 results) per dataset and on average; p95 query latency (95% of queries finish faster than this)BEIR (Thakur et al. 2021) BM25 baseline; Pyserini BM25 · measured locally—★★☆☆☆
PL-I09.GCN on ogbn-arxiv without PyG
MLPYTHONafter OGB node property prediction leaderboard
INTERMEDIATEtest accuracy (mean ± std over 10 seeds) on the official splitOGB leaderboard: GCN 0.7174 ± 0.0029 (OGB team), GraphSAGE 0.7149—★★☆☆☆
PL-I10.Logit lens vs tuned lens: reading GPT-2's guesses layer by layer
MLPYTHONafter Belrose et al. + 1
INTERMEDIATEper-layer KL(final ‖ lens), that is, how far each lens's prediction is from the model's real final prediction, on 1M held-out tokenstrained lenses from the AlignmentResearch tuned-lens package · measured locally—★★☆☆☆
PL-I11.Grokking, reverse-engineered: the Fourier circuit and beyond
MLPYTHONafter Nanda et al. + 1
INTERMEDIATEgrokking reproduced (test acc > 99% long after train acc reaches 100%); variance explained by the key frequencies; restricted/excluded-loss curvesNanda et al. 2023 progress measures · reproduced locally—★★★☆☆
PL-A01.Contamination detective: membership inference on open LLMs
MLPYTHONafter Shi et al. + 2
ADVANCEDAUROC (how well a score separates training members from non-members; 0.5 is chance, 1.0 is perfect) on WikiMIA at text lengths 32/64/128, per method and model; overlap rate per benchmark splitMin-K%++ paper (beats the runner-up by 6.2–10.5 AUROC pts on WikiMIA) · reproduced locally—★★★☆☆
PL-A02.LoRA + DPO from scratch, tested against TRL
MLPYTHONafter Rafailov et al. + 2
ADVANCEDheld-out preference accuracy (how often the implicit reward margin, the model's preference for the chosen over the rejected answer, is > 0); win rate vs SFT under a fixed reward model; peak GPU memoryTRL DPOTrainer + PEFT LoRA (same config) · measured locally—★★☆☆☆
PL-A03.Othello-GPT: find the board, then edit it
MLPYTHONafter Neel Nanda + 1
ADVANCEDlegal-move top-1 rate (how often the model's top predicted move is legal); per-layer linear-probe error; % of interventions whose new legal-move set matches the edited boardLi et al. 2023: linear probe 20.4% error vs non-linear 1.7% (per Nanda's write-up); Nanda's 'mine/theirs' linear probe—★★★☆☆
PL-A04.Sparse autoencoder on GPT-2, then steer with its features
MLPYTHONafter Bloom + 2
ADVANCEDCE loss recovered at matched L0; sentiment-classifier shift vs perplexity increase on 500 promptsJoseph Bloom's open GPT-2-small residual SAEs (24,576 latents, via SAELens) and ActAdd · measured locally—★★★☆☆
PL-A05.Ventilator: reverse-engineer the PID controller
MLPYTHONafter Kaggle + 1
ADVANCEDKaggle private-leaderboard MAE (mean absolute error), scored on the inspiratory (breathing-in) phase only; % of test breaths reconstructed exactlyGoogle Brain Ventilator Pressure Prediction private LB (leaderboard); the 1st of 2,650 teams used PID matching + LSTM/CNN-transformer—★★★☆☆
PL-A06.CIFAR-10 to 94% in seconds: rebuild airbench
MLPYTHONafter Keller Jordan + 2
ADVANCEDmean test accuracy over 25 runs and the wall-clock time to finish on your GPUairbench94_muon: 94.01% in 2.59 s on an A100 (README); run it on your own GPU to get the reference time—★★★☆☆
PL-A07.CLIP from scratch: port the weights, then train your own
MLPYTHONafter Radford et al. + 1
ADVANCEDcosine similarity between your embeddings and the reference embeddings; zero-shot top-1 (CIFAR-10, ImageNet-V2); zero-shot accuracy of your from-scratch modelthe openai/CLIP package with ViT-B/32 weights · measured locally—★★☆☆☆
PL-A08.DDPM vs flow matching on a 12-GPU-hour budget
MLPYTHONafter Ho et al. + 2
ADVANCEDFID-50k (Fréchet Inception Distance over 50k generated images; lower means more realistic and varied) vs number of function evaluations (NFE, the number of network calls one sample takes), at a fixed 12 GPU-hour training budgetDDPM paper: FID 3.17 on CIFAR-10 (with a much larger budget); pytorch-fid for scoring—★★★☆☆
PL-A09.AlphaZero Connect-4, graded by a perfect solver
MLPYTHONafter Silver et al. + 1
ADVANCED% optimal moves on 1,000 solver-labelled positions; win rate as first player vs the solver; GPU-hours usedPascal Pons's Connect-4 perfect solver (ground truth)—★★★☆☆
PL-A10.GPT-2 124M trainer with your own data-parallel + ZeRO-1
MLPYTHONafter Karpathy + 1
ADVANCEDFineWeb validation loss vs tokens seen; tokens/s scaling efficiency from 1→N GPUs; optimiser memory per GPUtorch DDP (measured locally); llm.c GPT-2 124M reproduction (90 min on 8×A100, ~$20)—★★★☆☆
PL-A11.Triton kernels on the roofline, up to FlashAttention
MLPYTHONafter Triton tutorial + 2
ADVANCEDattention TFLOP/s at sequence length 4k, head-dim 64, fp16 (forward and backward); LayerNorm's % of peak memory bandwidthPyTorch scaled_dot_product_attention (SDPA, flash backend) and the Triton fused-attention tutorial · measured locally—★★★☆☆
PL-A12.A tiny deep-learning compiler: trace, fuse, generate code
MLPYTHONafter JAX + 2
ADVANCEDforward+backward speedup over eager mode on a GPT-2 block and an MLP; kernel count; peak memorytorch.compile (Inductor backend) on the same block · measured locally—★★★☆☆
PL-A13.LLM serving engine: paged KV, continuous batching, int8
MLPYTHONafter Kwon et al. + 1
ADVANCEDoutput tokens/s; p50/p99 time-to-first-token (TTFT, the delay before the first token streams back) under load; int8 perplexity delta on WikiText-2vLLM on the same GPU · measured locally (nano-vllm README: 1434 vs vLLM 1362 tok/s on an RTX 4070 Laptop)—★★★☆☆
PL-W01.Chess-GPT: probe board and skill, then find something new
MLPYTHONafter Karvonen + 2
WOWboard-probe accuracy (best layer); skill-intervention win-rate multiplier (how many times more often the model wins after the intervention); new-latent probe accuracy plus causal effect sizeKarvonen 2024: board probe 99.6% (16-layer) / 99.1% (8-layer); the skill vector raises win rate up to ~2.6×—★★★★☆
PL-W02.Scaling scaling laws: Hex, then a game nobody fitted
MLPYTHONafter Andy L. Jones + 1
WOWextrapolation error of the 9×9 compute frontier when it is predicted from fits on boards ≤ 7×7; fit quality on the new gameJones 2021, Scaling Scaling Laws with Board Games (frontiers are exponential in performance and board size)—★★★★☆
PL-W03.BabyLM Strict-Small: beat the baseline on 10M words
MLPYTHONafter BabyLM Challenge + 1
WOWbabylm-eval aggregate score (Strict-Small) and the per-task breakdownOfficial BabyLM 2026 Strict-Small baselines and the Hugging Face leaderboard—★★★★☆
PL-W04.Vesuvius Challenge: ink detection for a Progress Prize
MLPYTHONafter Vesuvius Challenge + 1
WOWlate-submission score on the Kaggle Vesuvius Ink Detection private LB (leaderboard), using the competition metric; an accepted prize submissionKaggle Vesuvius Challenge Ink Detection (2023, 1,249 teams); scrollprize.org Progress Prizes ($250–$20,000 monthly, $20k best-of-month)—★★★★★
PL-W05.Tiny Recursive Model on ARC-AGI: reimplement, then improve
MLPYTHONafter Jolicoeur-Martineau + 2
WOWARC-AGI-1 and ARC-AGI-2 public-eval accuracy (pass@2: a task counts as solved if either of two allowed guesses is right)Jolicoeur-Martineau 2025 TRM: 45% ARC-AGI-1, 8% ARC-AGI-2 with ~7M params—★★★★★
PL-W06.Set a modded-nanogpt world record
MLPYTHONafter KellerJordan/modded-nanogpt + 1
WOWwall-clock time (or steps, on Track 3) to reach the target val loss on 8×H100, with p < 0.01 over runsmodded-nanogpt records: Track 1 #92 at 0.665 min (2026-08-30); Track 2 #18 at 17.35 min (2025-12-31)—★★★★★

INDUSTRY is the library, paper or record you measure against. MINE is your latest recorded result. IMPACT is a harsh hiring-manager rating of a finished version.