Brief
On tables of mixed numbers and categories, gradient-boosted trees are still the model to beat. XGBoost, LightGBM and scikit-learn's HistGradientBoostingClassifier all build on the same idea. In this project you write a small version yourself, reusing the tree from BL-05, and test it on a real prediction task.
The data is the Adult dataset (also called Census Income) from the UCI Machine Learning Repository. It has about 48,000 people from the 1994 US census, with columns such as age, education, occupation and hours worked per week, and a label saying whether their income was above 50,000 dollars a year. About a quarter earn above 50K, so always compare your accuracy with the 76% you would get by predicting "no" for everyone.
Boosting builds the model one tree at a time. The model's score for a row is a sum, , where each is a small tree and is the learning rate. The probability is . Each new tree is fitted to the negative gradient of the log-loss with respect to the score, which turns out to be simply : how far each prediction still is from the label. Leaf values come from one Newton step,
which divides the gradient by the curvature of the loss and makes each tree's step the right size.
DATA · a public tabular dataset
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your booster's test accuracy is within 2 percentage points of sklearn HistGradientBoostingClassifier on the same split. Record it with `python tools/mlarena.py record BL-09 "<your accuracy>"`.
- Training log-loss goes down every round, and the number of trees is chosen on the validation set, not the test set.
- The README compares your accuracy with the base rate and with your BL-05 single tree, and shows the feature-importance plot.
Side quests
Terms in plain words
- Gradient boosting
- Building a model as a sum of many small trees, where each new tree is trained to fix the errors the sum so far still makes.
- Learning rate (shrinkage)
- The factor, such as 0.1, that each new tree's output is multiplied by before it's added. Smaller steps need more trees but overfit less.
- Log-odds
- log(p / (1 − p)). The booster adds trees in this unbounded space and turns the total into a probability with the sigmoid.
- Histogram binning
- Replacing each feature's values with one of about 256 bin numbers, so the split search scans bins instead of every distinct value.
- Early stopping
- Stopping when the validation loss hasn't improved for a set number of rounds, and keeping the best round's trees.