0/ 38 SOLVED
BEGINNER0/10
INTERMEDIATE0/11
ADVANCED0/9
WOW0/8
0 wonML-ARENA BENCHMARKS
StatusPROJECTLEVELBENCHMARKINDUSTRYMINEIMPACT
PM-B02.How short can a git hash be? Birthday bounds tested on real repos
MATHPYTHONafter git-config documentation
BEGINNERHow far the predicted shortest unique prefix length is from the observed one, |predicted - observed|; and how well the analytic collision probability agrees with the simulated onegit's own choice under core.abbrev=auto · measured locally—★☆☆☆☆
PM-B03.Project the bias out of GloVe, then prove it is still there
MATHPYTHONafter Bolukbasi et al. 2016 + 3FITS NOW
BEGINNERWEAT effect size (career/family x male/female word sets), analogy accuracy, and k-means bias-recoverability accuracy (how well k-means clustering on the debiased vectors can still sort words by their original gender bias)Caliskan et al. 2017 WEAT protocol; Gonen & Goldberg 2019 recoverability test · numbers measured locally—★★☆☆☆
PM-B04.Finite differences vs the complex step: hunting the optimal h
MATHPYTHONafter Squire & Trapp 1998
BEGINNERRelative derivative error at the predicted-optimal h, and predicted vs measured optimal hSquire & Trapp 1998 complex-step method; IEEE fp64 eps = 2.2e-16 · measured locally—★☆☆☆☆
PM-B05.Gaussian elimination that breaks on purpose
MATHPYTHONafter originalFITS NOW
BEGINNERNormwise backward error ||Ax - b|| / (||A|| ||x||) on 4 matrix families, n = 50..1000 (roughly, how much you would have to change the problem for your answer to be exact); growth factor (how much the entries grow during elimination)numpy.linalg.solve (LAPACK getrf/getrs) · measured locally—★★☆☆☆
PM-B06.PageRank of the Python standard library
MATHPYTHONafter originalFITS NOW
BEGINNERL1 distance to the reference ranking, iterations to reach 1e-10, wall timenetworkx.pagerank · measured locally—★☆☆☆☆
PM-B07.Error bars for evals: when is 'model A beats model B' real?
MATHPYTHONafter Evan Miller 2024
BEGINNEREmpirical coverage of nominal 95% intervals across 40 simulated settings; minimum detectable difference (the smallest accuracy gap the test can reliably tell apart from noise) for a real model pairMiller 2024, Adding Error Bars to Evals (arXiv 2411.00640) · coverage measured locally—★★★☆☆
PM-B08.Bayesian ranking: sort Stack Exchange posts without being fooled by one vote
MATHPYTHONafter Evan Miller + 1
BEGINNERKendall tau (a rank correlation where +1 means the two orders agree perfectly) between the early-vote ranking and the final ranking, on 3 sitesRaw-average and Wilson-bound ranking (Evan Miller) · measured locally—★★☆☆☆
PM-B09.Why Euler throws the planet into the Sun
MATHPYTHONafter original
BEGINNERRelative energy error after 1000 orbits, at an equal number of force evaluations; how long the figure-eight survivesscipy.integrate.solve_ivp (RK45, DOP853) · measured locally—★★☆☆☆
PM-B10.Optimizer golf on nasty functions
MATHPYTHONafter Gabriel Goh
BEGINNERFunction+gradient evaluations to reach f - f* < 1e-8; predicted vs measured linear rates (the factor by which the error shrinks each step)scipy.optimize.minimize (BFGS) · measured locally—★★☆☆☆
PM-B11.A hand-written FFT that denoises a real recording
MATHPYTHONafter Jez Swanson
BEGINNERMax abs error vs numpy.fft and wall time at n = 2^20 complex points; FFT-convolution speedupnumpy.fft (pocketfft) · measured locally—★★☆☆☆
PM-I01.Break RANDU, build PCG32, sample anything
MATHPYTHONafter PCG + 2
INTERMEDIATEBattery p-values per generator; relative error of the P(Z > 6) estimate at a fixed sample budget (Z is a standard normal variable)TestU01 batteries (L'Ecuyer & Simard) and NumPy's PCG64 · measured locally—★★☆☆☆
PM-I02.Gauss vs Clenshaw-Curtis: a quadrature shootout
MATHPYTHONafter Trefethen 2008 + 1
INTERMEDIATEFunction evaluations needed to reach 1e-10 relative error, per Genz familyscipy.integrate.quad (QUADPACK) · measured locally—★★☆☆☆
PM-I03.Gram-Schmidt vs Householder: watching orthogonality die
MATHPYTHONafter NIST Statistical Reference DatasetsFITS NOW
INTERMEDIATE||Q^T Q - I|| vs kappa (how far the computed Q is from orthogonal); log relative error (correct digits) against NIST certified valuesNIST StRD certified results; numpy.linalg.lstsq (LAPACK) · measured locally—★★☆☆☆
PM-I04.Randomized SVD: 95% of the answer at 5% of the cost
MATHPYTHONafter HalkoFITS NOW
INTERMEDIATE||A - QQ^T A||_2 / sigma_(k+1) (your approximation error divided by the best error any rank-k approximation can reach) and wall time at k = 100sklearn.utils.extmath.randomized_svd and scipy.sparse.linalg.svds (ARPACK) · measured locally—★★☆☆☆
PM-I05.Is it really a power law? Clauset-grade fitting
MATHPYTHONafter Clauset + 2
INTERMEDIATEalpha-hat (fitted exponent), x_min-hat (fitted cutoff) and accept/reject verdicts vs the paper's 24 datasetsClauset, Shalizi & Newman 2009 (arXiv 0706.1062) and the `powerlaw` package · measured locally—★★★☆☆
PM-I06.Find the Ising critical temperature with your own Monte Carlo
MATHPYTHONafter Square lattice Ising model
INTERMEDIATE|T_c estimate - exact| / exact, with a 2-sigma error bar; dynamic exponent z (how fast the autocorrelation time grows with grid size)Onsager exact kT_c/J = 2/ln(1 + sqrt 2) ~ 2.26919 (Wikipedia)—★★☆☆☆
PM-I07.Kalman filters that pass their own consistency test
MATHPYTHONafter Roger Labbe + 2
INTERMEDIATEPosition RMSE; fraction of runs whose average NEES (normalised estimation error squared) falls inside the 95% chi-square bandfilterpy (Labbe) · measured locally—★★☆☆☆
PM-I08.Conjugate gradient on a million-unknown Poisson problem
MATHPYTHONafter Jonathan Shewchuk + 1FITS NOW
INTERMEDIATEIterations and wall time to reach relative residual 1e-8 (||b - Ax|| / ||b||)scipy.sparse.linalg.cg and spsolve (SuperLU) · measured locally—★★☆☆☆
PM-I09.Arithmetic coding: land within 0.1% of the entropy you predicted
MATHPYTHONafter Matt Mahoney
INTERMEDIATECompressed bytes on enwik7; overhead = (actual bits - model cross-entropy)gzip -9 / bzip2 -9 / xz -9 on the same file · measured locally (LTCB, the Large Text Compression Benchmark, full-enwik8 reference: gzip 36,445,248; bzip2 29,008,736; xz 24,703,772 bytes)—★★☆☆☆
PM-I10.Gaussian process from scratch on 60 years of Mauna Loa CO2
MATHPYTHONafter Görtler + 3
INTERMEDIATEHeld-out RMSE and 95%-interval coverage on the last 10 years; optimised log marginal likelihood (LML)scikit-learn GaussianProcessRegressor CO2 example · measured locally—★★☆☆☆
PM-I11.Tensor autodiff in both directions, with linear-algebra gradients
MATHPYTHONafter Max Slater + 2
INTERMEDIATEMax relative gradient/HVP error vs PyTorch float64 on a 40-function suite; reverse-mode overhead vs PyTorch eager (how much slower your library runs)PyTorch / JAX in float64 · measured locally—★★★☆☆
PM-A01.Newton-Schulz polynomials for the Muon optimizer: beat the default coefficients
MATHPYTHONafter Keller Jordan + 1
ADVANCEDmax_i |sigma_i(X_5) - 1| (the worst distance of any singular value from 1 after 5 steps) and ||X_5^T X_5 - I||_F (how far X_5 is from orthogonal) in emulated bf16 on 50 real gradient matrices; validation loss of a toy runMuon default coefficients (Keller Jordan) and Polar Express (arXiv 2505.16932) · measured locally—★★★☆☆
PM-A02.Which Chatbot Arena gaps are real? Bradley-Terry with honest uncertainty
MATHPYTHONafter Chiang et al. 2024 + 2
ADVANCEDMLE agreement with a reference solver; interval coverage in simulation; fraction of adjacent pairs separable at 95%Chatbot Arena paper methodology (arXiv 2403.04132) · measured locally—★★★☆☆
PM-A03.Compress GPT-2 with activation-aware SVD, in pure NumPy
MATHPYTHONafter Jay Mody + 2FITS NOW
ADVANCEDWikiText-2 perplexity (how surprised the model is by real text; lower is better) at 20% and 40% parameter reductionSVD-LLM (arXiv 2403.07378) and ASVD (arXiv 2312.05821) methods; GPT-2 numbers measured locally—★★★☆☆
PM-A04.Beat CPython's big-int multiply with an FFT you can trust
MATHPYTHONafter CPython Lib/_pylong.py + 2
ADVANCEDTime to multiply two 1M-digit and two 10M-digit integers; time to 10M digits of piCPython int (Karatsuba, KARATSUBA_CUTOFF = 70 in longobject.c) and gmpy2 (GMP) · measured locally—★★★☆☆
PM-A05.Neural-ODE adjoint from scratch, and where it lies
MATHPYTHONafter Chen et al. 2018 + 1
ADVANCEDGradient relative error vs backprop-through-solver at matched tolerance; peak memory vs number of stepstorchdiffeq odeint_adjoint (Chen et al. 2018) · measured locally—★★★☆☆
PM-A06.Lanczos on a two-million-node road network
MATHPYTHONafter SNAP + 1FITS NOW
ADVANCEDWall time and max residual ||Lv - lambda v|| for k = 100; edge cut of spectral bisection (the number of edges crossing between the halves)scipy.sparse.linalg.eigsh (ARPACK) and METIS edge cut (METIS is the standard graph-partitioning tool) · measured locally—★★★☆☆
PM-A07.2D Gaussian splatting with every gradient derived by hand
MATHPYTHONafter Zhang et al. 2024 + 1
ADVANCEDMean PSNR (peak signal-to-noise ratio, an image-quality score in dB; higher is better) over the 24 Kodak images at 30,000 Gaussians; finite-difference gradient checkGaussianImage (arXiv 2403.08551) Kodak results at the same Gaussian count (GPU) · compared against their reported tables—★★★☆☆
PM-A08.An interior-point LP solver vs the whole Netlib suite
MATHPYTHONafter Netlib LP test problems + 1
ADVANCEDNumber of Netlib problems solved to 1e-6 relative objective error; median IPM iterationsHiGHS via scipy.optimize.linprog(method='highs-ipm') · measured locally; optimal values from the Netlib lp/data readme—★★★☆☆
PM-A09.NUTS (the No-U-Turn Sampler) from scratch, audited by simulation-based calibration
MATHPYTHONafter Hoffman & Gelman 2014 + 3
ADVANCEDPosterior mean/sd error in MCSE units (Monte Carlo standard error: the expected noise from using a finite number of draws) vs posteriordb reference draws; ESS (effective sample size: how many independent draws your correlated draws are worth) per gradient evaluationStan (CmdStan) on the same posteriors · measured locally; posteriordb reference posteriors—★★★★☆
PM-W01.Land an accuracy fix in SciPy's special functions
MATHPYTHONafter mpmath + 2
WOWMax relative error (ULPs) on the affected domain, before vs after the fixmpmath 50-digit reference vs the current SciPy release · measured locally—★★★★☆
PM-W02.PageRank the web on one machine: Common Crawl's domain graph
MATHPYTHONafter Common Crawl web graphs + 1FITS NOW
WOWSpearman rho (a rank correlation where 1 means identical order) vs Common Crawl's published domain PageRank (top 1M); wall time; peak RSS (the most RAM the process used at any moment)Common Crawl domain-ranks file (computed with the WebGraph framework) · runtime measured locally—★★★☆☆
PM-W03.Hunt a Packomania or AlphaEvolve packing record
MATHPYTHONafter Packomania + 4
WOWObjective value vs the current Packomania record or published AlphaEvolve value for each nPackomania records (E. Specht); AlphaEvolve report (arXiv 2506.13131), ~2.635 sum of radii for n = 26—★★★★☆
PM-W04.LLM as compressor: GPT-2 plus your arithmetic coder vs cmix
MATHPYTHONafter Deletang et al. 2023 + 2
WOWBits per byte on the 10 MB prefix; total size including decompressor and weightscmix v21 on the same prefix · measured locally (LTCB full enwik8: cmix v21 14,623,723; nncp v3.2 14,915,298 bytes); Deletang et al. 2023, Language Modeling Is Compression—★★★★☆
PM-W05.Decimeter Challenge: smartphone GNSS with estimators you built
MATHPYTHONafter Kaggle + 1
WOWOfficial competition metric on the private leaderboard (late submission)GSDC 2022 final private leaderboard; winning team's two-step optimisation write-up (PMC9919037)—★★★★☆
PM-W06.Exact Gaussian process on hundreds of thousands of points, on a CPU
MATHPYTHONafter Wang et al. 2019 + 1
WOWTest RMSE and NLL (negative log-likelihood of the test data, which also scores the uncertainty estimates) vs the exact-GP results in Wang et al. 2019; wall time on CPU; log-det estimate errorWang et al. 2019 (arXiv 1903.08114) multi-GPU results; GPyTorch · measured locally—★★★★☆
PM-W07.Muon at the edge of stability: a sharpness study with your own autodiff
MATHPYTHONafter Cohen et al. 2021 + 3
WOWMeasured sharpness vs 2/eta (GD) and vs the preconditioned-sharpness threshold (Adam); Lanczos top-eigenvalue error vs a dense Hessian on a tiny modelCohen et al. 2021 (arXiv 2103.00065); Cohen et al. 2022 adaptive EoS (arXiv 2207.14484); PyHessian · measured locally—★★★★☆
PM-W08.Flatten Herculaneum scrolls: low-distortion parametrisation for VC3D
MATHPYTHONafter Vesuvius Challenge prizes + 2FITS NOW
WOWIsometric distortion (symmetric Dirichlet / L2 stretch: how much lengths are stretched or squashed), angle distortion, and runtime on full segmentsOpenABF (ABF++/LSCM) and current VC3D flattening · measured locally; Vesuvius Challenge prize criteria ('low-distortion isometric 2D parametrization')—★★★★☆

INDUSTRY is the library, paper or record you measure against. MINE is your latest recorded result. IMPACT is a harsh hiring-manager rating of a finished version.