← All Base Form Math projects
BM-30NOT STARTEDBASE FORM · PROBABILITYMATHPYTHON

Smart autotuner

Tune BM-02's block size (or any slow setting) with a Gaussian process that picks the next experiment for you.

EST.10 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
GP prediction; experiments to find the best
GOAL · match within 1e-6; fewer runs than grid
sklearn GaussianProcessRegressor · grid search—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Some settings can only be tuned by trying them, and each try is slow. Grid search tests every option. A smarter tuner remembers what it has seen, guesses what the untried options would do, and spends the next experiment where it is most likely to help. In this project you build that tuner with a Gaussian process (GP) and point it at the block size from BM-02.

A GP is a prior over functions. You pick a kernel, here the RBF kernel k(x,x′)=σf2exp⁡(−(x−x′)2/2ℓ2)k(x, x') = \sigma_f^2 \exp\left(-(x - x')^2 / 2\ell^2\right), which says nearby inputs give similar outputs. After seeing data (X,y)(X, \mathbf{y}) with noise variance σn2\sigma_n^2, the prediction at a new point x∗x_* is normal with mean k∗⊤(K+σn2I)−1y\mathbf{k}_*^\top (K + \sigma_n^2 I)^{-1}\mathbf{y} and variance k(x∗,x∗)−k∗⊤(K+σn2I)−1k∗k(x_*, x_*) - \mathbf{k}_*^\top (K + \sigma_n^2 I)^{-1}\mathbf{k}_*. That is a linear system, solved with a Cholesky factorisation rather than an explicit inverse.

The GP gives a mean and an uncertainty everywhere. Expected improvement turns both into one score: high where the mean looks good, and also high where the GP is unsure and something good might be hiding. Run the best-scoring setting, add the result, refit, and repeat. Use the base-2 log of the block size as the input, since 8 to 16 is as big a step as 256 to 512.

VISUALFig. 1
log run time: GP mean ± 2 sd, dashed = truth next run expected improvement 8 16 32 64 128 256 512 block size (log scale)
A GP after four timed block sizes. The band is widest between measurements, and expected improvement peaks in the gap where a faster setting could still be. Why it's here: Shows how the tuner trades off a good mean against high uncertainty when it picks the next run.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.

DATA · your own benchmarks

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your GP mean and sd match sklearn's GaussianProcessRegressor within 1e-6, and the tuner reaches the best setting in fewer runs than the grid. Record it with `python tools/mlarena.py record BM-30 "<GP runs>/<grid runs>"`.
  2. Over 20 seeds, the GP finds a setting within 2% of the grid's best using fewer runs on average than random search.
  3. A plot sequence shows the GP mean, band and expected improvement after each new run.

Side quests

Tune two settings at once, such as block size and repeat count or a thread count, and compare the run savings with the 1D case.
Swap the RBF kernel for a Matérn 5/2 kernel and see whether rougher assumptions help on noisy timings.
Run the same problem through Optuna's default sampler and compare runs needed.

Terms in plain words

Gaussian process (GP)
A distribution over functions. Any finite set of function values is jointly normal, with covariances given by a kernel.
Kernel
A function k(x, x′) that says how similar the outputs at two inputs should be. The RBF kernel makes nearby inputs give similar outputs.
Length scale
The RBF kernel's distance setting. Inputs much closer than it are nearly identical; inputs much further apart are nearly unrelated.
Marginal likelihood
The probability of the observed data under the GP with given kernel settings. Maximising it picks the length scale and noise.
Expected improvement
An acquisition function. For each candidate, the average amount by which it would beat the best result so far, counting uncertainty.