Brief
Some settings can only be tuned by trying them, and each try is slow. Grid search tests every option. A smarter tuner remembers what it has seen, guesses what the untried options would do, and spends the next experiment where it is most likely to help. In this project you build that tuner with a Gaussian process (GP) and point it at the block size from BM-02.
A GP is a prior over functions. You pick a kernel, here the RBF kernel , which says nearby inputs give similar outputs. After seeing data with noise variance , the prediction at a new point is normal with mean and variance . That is a linear system, solved with a Cholesky factorisation rather than an explicit inverse.
The GP gives a mean and an uncertainty everywhere. Expected improvement turns both into one score: high where the mean looks good, and also high where the GP is unsure and something good might be hiding. Run the best-scoring setting, add the result, refit, and repeat. Use the base-2 log of the block size as the input, since 8 to 16 is as big a step as 256 to 512.
DATA · your own benchmarks
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your GP mean and sd match sklearn's GaussianProcessRegressor within 1e-6, and the tuner reaches the best setting in fewer runs than the grid. Record it with `python tools/mlarena.py record BM-30 "<GP runs>/<grid runs>"`.
- Over 20 seeds, the GP finds a setting within 2% of the grid's best using fewer runs on average than random search.
- A plot sequence shows the GP mean, band and expected improvement after each new run.
Side quests
Terms in plain words
- Gaussian process (GP)
- A distribution over functions. Any finite set of function values is jointly normal, with covariances given by a kernel.
- Kernel
- A function k(x, x′) that says how similar the outputs at two inputs should be. The RBF kernel makes nearby inputs give similar outputs.
- Length scale
- The RBF kernel's distance setting. Inputs much closer than it are nearly identical; inputs much further apart are nearly unrelated.
- Marginal likelihood
- The probability of the observed data under the GP with given kernel settings. Maximising it picks the length scale and noise.
- Expected improvement
- An acquisition function. For each candidate, the average amount by which it would beat the best result so far, counting uncertainty.