Brief
In NumPy, implement multinomial (softmax, multi-class) logistic regression with L2, an analytic gradient, and Hessian-vector products. L2 here is a penalty on the squared weights; a Hessian-vector product multiplies the matrix of second derivatives by a vector without ever building the full matrix. Write your own L-BFGS solver (a quasi-Newton method that estimates curvature from the last few gradient steps) and your own Newton-CG solver (Newton's method where each step is solved approximately with conjugate gradients, using only Hessian-vector products). Give both a line search (a rule that decides how far to move along each step direction). Your solver must reach the same optimum as scikit-learn's lbfgs on full Covertype (581,012 rows, 54 features, 7 classes). Then report calibration, meaning whether predicted probabilities match how often the model is actually right, with ECE (expected calibration error) and a reliability diagram. Finally, apply temperature scaling (divide the logits by one learned number to fix over- or under-confidence).
VICTORY CONDITIONS
- log-loss within 1e-4 of sklearn at equal C (the same regularisation strength), in ≤ 1.0× sklearn's wall-clock; ECE ≤ 0.02 after temperature scaling (the gap between your final training log-loss and sklearn's, and the wall-clock time to reach it (single thread); test ECE).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- L-BFGS
- An optimiser that approximates Newton's method by remembering the last few gradient changes instead of storing the full Hessian.
- Newton-CG
- Newton's method where each step is found approximately by conjugate gradients, which only needs Hessian-vector products.
- Hessian-vector product
- The second-derivative matrix times a vector, computed directly without building the matrix.
- ECE (expected calibration error)
- The average gap between the model's stated confidence and its actual accuracy, measured over confidence buckets.
- Temperature scaling
- Dividing all logits by one tuned number so the predicted probabilities become better calibrated.
Why it rates ★☆☆☆☆
Logistic regression from scratch is expected knowledge. Beating sklearn's time is a nice extra, but it is not a strong signal.