← All Cracked ML projects
PL-I03NOT STARTEDINTERMEDIATEMLPYTHONIMPACT ★★☆☆☆

Histogram GBDT in NumPy vs LightGBM on 1M HIGGS rows

Leaf-wise, second-order, histogram-subtraction gradient boosting in vectorised NumPy, raced against LightGBM.

EST.16 h
REWARD+300 XP
AFTERKe et al. + 1

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
test AUC and training wall-clock at equal hyper-parameters (num_leaves=255, 500 rounds, lr=0.1)
GOAL · AUC within 0.003 of LightGBM at ≤ 10× its wall-clock
LightGBM (single thread) · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Quantile-bin every feature into 255 bins, so each bin holds about the same number of rows. Build gradient/hessian histograms with np.bincount: for each bin, sum the first and second derivatives of the loss over the rows in it. Use the parent-minus-sibling histogram trick: build the histogram for the smaller child only and get the other child by subtracting it from the parent. Grow trees leaf-wise (always split the leaf with the biggest gain, rather than one whole level at a time) using L2-regularised second-order gain (the XGBoost-style split score built from gradient and hessian sums), min_child_weight (a minimum hessian sum per leaf) and NaN routing (learn which side of each split missing values go to). Train on 1M HIGGS rows and test on the official last-500k split.

VICTORY CONDITIONS

  1. AUC within 0.003 of LightGBM at ≤ 10× its wall-clock (test AUC and training wall-clock at equal hyper-parameters (num_leaves=255, 500 rounds, lr=0.1)).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

GBDT
Gradient-boosted decision trees: each new tree is fitted to correct the errors of all the trees before it.
Histogram binning
Replacing each feature value with a bin number so split search only has to scan 255 bins instead of every unique value.
Second-order gain
A split score that uses both the gradient and the hessian (curvature) of the loss, as XGBoost and LightGBM do.
Leaf-wise growth
Always splitting whichever leaf improves the loss most, which gives deeper, uneven trees.
AUC
The probability that the model ranks a random positive example above a random negative one.

Why it rates ★★☆☆☆

GBDTs (gradient-boosted decision trees) still win on tabular data and few people can build one. Matching LightGBM's AUC is credible.