← All Cracked ML projects
PL-B02NOT STARTEDBEGINNERMLPYTHONIMPACT ★★☆☆☆

Reproduce double descent with random features

Make test error go up and then down again as you add parameters. Do it on purpose, on a real dataset.

EST.5 h
REWARD+200 XP
AFTERBelkin + 1

ML-ARENA BENCHMARKS

PROOF: FINDING
METRICINDUSTRYMINE
where the test-MSE peak sits and how high it is; best over-parameterised (N > n) test error vs best under-parameterised (N < n) test error
GOAL · peak within N ∈ [0.9n, 1.1n]; error at N=20n below the best classical (N<n) error; the tuned ridge curve is monotone (it only goes down, with no peak)
The double-descent curve shape from Belkin et al. 2019 (PNAS) · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In NumPy, fit min-norm least squares on random Fourier features of a fixed n=2000 subset of Covertype, with one-hot targets. Random Fourier features are random cosine transforms of the input; they let you build a linear model with as many features as you like. Min-norm least squares means that when many weight vectors fit the training data exactly, you pick the one with the smallest length (this is what the pseudo-inverse gives you). Sweep the feature count N from 0.1n to 20n. Then repeat with a sweep over the ridge penalty λ (an L2 penalty on the weights). The plot must show the interpolation peak at N≈n, the point where the model has just enough features to fit the training data exactly and test error spikes. It must also show that tuned ridge removes that peak.

VICTORY CONDITIONS

  1. peak within N ∈ [0.9n, 1.1n]; error at N=20n below the best classical (N<n) error; the tuned ridge curve is monotone (it only goes down, with no peak) (where the test-MSE peak sits and how high it is; best over-parameterised (N > n) test error vs best under-parameterised (N < n) test error).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Double descent
Test error falls, rises to a peak when the model can just barely fit the training data, then falls again as you keep adding parameters.
Interpolation peak
The spike in test error at N≈n, where the model fits every training point exactly but in a very unstable way.
Min-norm least squares
Among all weight vectors that fit the data equally well, the one with the smallest length.
Random Fourier features
Random cosine features of the input that turn a linear model into a flexible non-linear one with any number of parameters.

Why it rates ★★☆☆☆

A clean reproduction of a famous phenomenon shows you really understand the bias-variance trade-off. But it is a known result.