Brief
Every project so far ended with one model and one score. Real work starts with a question: which model should I use, and can I trust its score? In this project you build a small AutoML tool that answers both. It runs every model you have built on a dataset, scores them with honest cross-validation, warns you about data leakage, and picks a winner.
Use datasets you already know from BL-01 to BL-09, such as the UCI Bike Sharing data, the SMS Spam Collection or the Adult census data. You know how each one should behave, which makes it easier to spot a score that's too good.
Cross-validation gives a better estimate than one split. With folds the score is the average , and the spread across folds tells you how much to trust it. Leakage is the main way this goes wrong. Common causes are a feature that is really part of the answer, scaling or imputing using all the data before splitting, the same person or duplicate row on both sides of the split, and random folds on data where time matters. Each one inflates the score and can't be seen in the score alone.
The tool you write here is the evaluation harness for the rest of the track. Keep it simple, and keep it strict.
DATA · any dataset from BL-01 to BL-09
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- For every model, your fold scores match sklearn cross_val_score on the same folds to within 1e-9, and your test script catches all 5 planted leaks with no false alarm on clean data. Record the number caught with `python tools/mlarena.py record BL-10 "<leaks caught>/5"`.
- Every preprocessing step is fitted inside the folds, and the test set is used exactly once.
- The tool runs end to end on at least two datasets from earlier projects and the README shows both reports.
Side quests
Terms in plain words
- k-fold cross-validation
- Split the data into k parts, train on k − 1 of them and score on the one left out, k times, then average the k scores.
- Data leakage
- Information about the answer reaching the model during training in a way it won't at prediction time. It makes scores look better than they will be.
- Pipeline
- Preprocessing and model chained into one object, so every step is fitted on the training folds only.
- Group k-fold
- Cross-validation that keeps all rows from the same group, such as one person or one store, in the same fold.
- Nested cross-validation
- An outer loop that scores the whole model-picking process and an inner loop that does the picking, so the winner's score isn't biased upward.