Brief
In NumPy, fit min-norm least squares on random Fourier features of a fixed n=2000 subset of Covertype, with one-hot targets. Random Fourier features are random cosine transforms of the input; they let you build a linear model with as many features as you like. Min-norm least squares means that when many weight vectors fit the training data exactly, you pick the one with the smallest length (this is what the pseudo-inverse gives you). Sweep the feature count N from 0.1n to 20n. Then repeat with a sweep over the ridge penalty λ (an L2 penalty on the weights). The plot must show the interpolation peak at N≈n, the point where the model has just enough features to fit the training data exactly and test error spikes. It must also show that tuned ridge removes that peak.
VICTORY CONDITIONS
- peak within N ∈ [0.9n, 1.1n]; error at N=20n below the best classical (N<n) error; the tuned ridge curve is monotone (it only goes down, with no peak) (where the test-MSE peak sits and how high it is; best over-parameterised (N > n) test error vs best under-parameterised (N < n) test error).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Double descent
- Test error falls, rises to a peak when the model can just barely fit the training data, then falls again as you keep adding parameters.
- Interpolation peak
- The spike in test error at N≈n, where the model fits every training point exactly but in a very unstable way.
- Min-norm least squares
- Among all weight vectors that fit the data equally well, the one with the smallest length.
- Random Fourier features
- Random cosine features of the input that turn a linear model into a flexible non-linear one with any number of parameters.
Why it rates ★★☆☆☆
A clean reproduction of a famous phenomenon shows you really understand the bias-variance trade-off. But it is a known result.