← All Cracked ML projects
PL-B03NOT STARTEDBEGINNERMLPYTHONIMPACT ★☆☆☆☆

Softmax regression speed duel on 581k rows

Write your own L-BFGS and Newton-CG optimisers and match scikit-learn's optimum to 4 decimals, faster.

EST.6 h
REWARD+200 XP
AFTERUCI Covertype dataset

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
the gap between your final training log-loss and sklearn's, and the wall-clock time to reach it (single thread); test ECE
GOAL · log-loss within 1e-4 of sklearn at equal C (the same regularisation strength), in ≤ 1.0× sklearn's wall-clock; ECE ≤ 0.02 after temperature scaling
scikit-learn LogisticRegression(solver='lbfgs') · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In NumPy, implement multinomial (softmax, multi-class) logistic regression with L2, an analytic gradient, and Hessian-vector products. L2 here is a penalty on the squared weights; a Hessian-vector product multiplies the matrix of second derivatives by a vector without ever building the full matrix. Write your own L-BFGS solver (a quasi-Newton method that estimates curvature from the last few gradient steps) and your own Newton-CG solver (Newton's method where each step is solved approximately with conjugate gradients, using only Hessian-vector products). Give both a line search (a rule that decides how far to move along each step direction). Your solver must reach the same optimum as scikit-learn's lbfgs on full Covertype (581,012 rows, 54 features, 7 classes). Then report calibration, meaning whether predicted probabilities match how often the model is actually right, with ECE (expected calibration error) and a reliability diagram. Finally, apply temperature scaling (divide the logits by one learned number to fix over- or under-confidence).

VICTORY CONDITIONS

  1. log-loss within 1e-4 of sklearn at equal C (the same regularisation strength), in ≤ 1.0× sklearn's wall-clock; ECE ≤ 0.02 after temperature scaling (the gap between your final training log-loss and sklearn's, and the wall-clock time to reach it (single thread); test ECE).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

L-BFGS
An optimiser that approximates Newton's method by remembering the last few gradient changes instead of storing the full Hessian.
Newton-CG
Newton's method where each step is found approximately by conjugate gradients, which only needs Hessian-vector products.
Hessian-vector product
The second-derivative matrix times a vector, computed directly without building the matrix.
ECE (expected calibration error)
The average gap between the model's stated confidence and its actual accuracy, measured over confidence buckets.
Temperature scaling
Dividing all logits by one tuned number so the predicted probabilities become better calibrated.

Why it rates ★☆☆☆☆

Logistic regression from scratch is expected knowledge. Beating sklearn's time is a nice extra, but it is not a strong signal.