← All Base Form ML projects
BL-09NOT STARTEDBASE FORM · CLASSICAL MLMLPYTHON

Tiny gradient booster

Build gradient-boosted trees from scratch and use them on a real tabular problem.

EST.9 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
test accuracy
GOAL · within 2%
sklearn HistGradientBoostingClassifier—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

On tables of mixed numbers and categories, gradient-boosted trees are still the model to beat. XGBoost, LightGBM and scikit-learn's HistGradientBoostingClassifier all build on the same idea. In this project you write a small version yourself, reusing the tree from BL-05, and test it on a real prediction task.

The data is the Adult dataset (also called Census Income) from the UCI Machine Learning Repository. It has about 48,000 people from the 1994 US census, with columns such as age, education, occupation and hours worked per week, and a label saying whether their income was above 50,000 dollars a year. About a quarter earn above 50K, so always compare your accuracy with the 76% you would get by predicting "no" for everyone.

Boosting builds the model one tree at a time. The model's score for a row is a sum, FM(x)=F0+η∑m=1Mhm(x)F_M(x) = F_0 + \eta \sum_{m=1}^{M} h_m(x), where each hmh_m is a small tree and η\eta is the learning rate. The probability is p=σ(F)p = \sigma(F). Each new tree is fitted to the negative gradient of the log-loss with respect to the score, which turns out to be simply yi−piy_i - p_i: how far each prediction still is from the label. Leaf values come from one Newton step,

vleaf=∑i∈leaf(yi−pi)∑i∈leafpi(1−pi),v_{\text{leaf}} = \frac{\sum_{i \in \text{leaf}} (y_i - p_i)}{\sum_{i \in \text{leaf}} p_i (1 - p_i)},

which divides the gradient by the curvature of the loss and makes each tree's step the right size.

OVERVIEWFig. 1
1 · SCOREp = σ(F)start: F = log-oddsof the base rate 2 · RESIDUALr = y − pwhat each row stillgets wrong 3 · FITsmall tree hsplits on r, Newtonstep in each leaf 4 · ADDF ← F + η·hη ≈ 0.1 keepseach step small REPEAT UNTIL VALIDATION LOSS STOPS FALLING
One boosting round. Score, take the residual, fit a small tree to it, add a shrunken copy, repeat. Why it's here: The whole algorithm is this loop. Objectives 2 and 3 build the two boxes in the middle.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.

DATA · a public tabular dataset

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your booster's test accuracy is within 2 percentage points of sklearn HistGradientBoostingClassifier on the same split. Record it with `python tools/mlarena.py record BL-09 "<your accuracy>"`.
  2. Training log-loss goes down every round, and the number of trees is chosen on the validation set, not the test set.
  3. The README compares your accuracy with the base rate and with your BL-05 single tree, and shows the feature-importance plot.

Side quests

Send missing values down whichever side of each split lowers the loss most, the way HistGradientBoostingClassifier does, and measure the effect.
Plot validation log-loss against the number of trees for learning rates 0.3, 0.1 and 0.03.

Terms in plain words

Gradient boosting
Building a model as a sum of many small trees, where each new tree is trained to fix the errors the sum so far still makes.
Learning rate (shrinkage)
The factor, such as 0.1, that each new tree's output is multiplied by before it's added. Smaller steps need more trees but overfit less.
Log-odds
log(p / (1 − p)). The booster adds trees in this unbounded space and turns the total into a probability with the sigmoid.
Histogram binning
Replacing each feature's values with one of about 256 bin numbers, so the split search scans bins instead of every distinct value.
Early stopping
Stopping when the validation loss hasn't improved for a set number of rounds, and keeping the best round's trees.