← All Base Form ML projects
BL-10NOT STARTEDBASE FORM · CLASSICAL MLMLPYTHON

AutoML-lite

A script that tries every model you've built, cross-validates them honestly, flags data leakage, and picks the winner.

EST.8 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
CV scores; leakage tests
GOAL · same scores; catches 5 planted leaks
sklearn cross_val_score—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Every project so far ended with one model and one score. Real work starts with a question: which model should I use, and can I trust its score? In this project you build a small AutoML tool that answers both. It runs every model you have built on a dataset, scores them with honest cross-validation, warns you about data leakage, and picks a winner.

Use datasets you already know from BL-01 to BL-09, such as the UCI Bike Sharing data, the SMS Spam Collection or the Adult census data. You know how each one should behave, which makes it easier to spot a score that's too good.

Cross-validation gives a better estimate than one split. With kk folds the score is the average CV=1k∑j=1kscorej\text{CV} = \frac{1}{k} \sum_{j=1}^{k} \text{score}_j, and the spread across folds tells you how much to trust it. Leakage is the main way this goes wrong. Common causes are a feature that is really part of the answer, scaling or imputing using all the data before splitting, the same person or duplicate row on both sides of the split, and random folds on data where time matters. Each one inflates the score and can't be seen in the score alone.

The tool you write here is the evaluation harness for the rest of the track. Keep it simple, and keep it strict.

VISUALFig. 1
ROUND 1 2 3 4 5 TESTTESTTESTTESTTEST EACH ROUND 1. fit on the training folds imputer, scaler, encoder, model 2. apply to the test fold transform only, never refit 3. score, then average report mean ± std over 5 rounds UNMARKED = TRAINING FOLD · TEST = HELD-OUT FOLD
Five-fold cross-validation. Each round fits the whole pipeline, preprocessing included, on four folds and scores on the fifth. Why it's here: Shows where preprocessing must be fitted. Fitting it before the split is one of the five leaks objective 5 plants.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.

DATA · any dataset from BL-01 to BL-09

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. For every model, your fold scores match sklearn cross_val_score on the same folds to within 1e-9, and your test script catches all 5 planted leaks with no false alarm on clean data. Record the number caught with `python tools/mlarena.py record BL-10 "<leaks caught>/5"`.
  2. Every preprocessing step is fitted inside the folds, and the test set is used exactly once.
  3. The tool runs end to end on at least two datasets from earlier projects and the README shows both reports.

Side quests

Add a permutation check. Shuffle the target, rerun the winner, and confirm its score drops to chance. If it doesn't, something is leaking.
Add random search over each model's main hyperparameter inside the inner loop and see whether the winner changes.

Terms in plain words

k-fold cross-validation
Split the data into k parts, train on k − 1 of them and score on the one left out, k times, then average the k scores.
Data leakage
Information about the answer reaching the model during training in a way it won't at prediction time. It makes scores look better than they will be.
Pipeline
Preprocessing and model chained into one object, so every step is fitted on the training folds only.
Group k-fold
Cross-validation that keeps all rows from the same group, such as one person or one store, in the same fold.
Nested cross-validation
An outer loop that scores the whole model-picking process and an inner loop that does the picking, so the winner's score isn't biased upward.