Brief
Write a CLI that takes a tabular dataset, a target column and a split spec (which rows go to train and which to test), then flags data leaks. A leak is information that would not be available at prediction time but sneaks into training and makes the model look better than it is. Your tool must catch: suspiciously predictive single features; post-outcome timestamps (features recorded after the event you are predicting); exact and near-duplicate rows across splits; group leakage (the same entity, such as one patient or one customer, in both train and test); and preprocessing fitted before the split (for example, a scaler fitted on all rows, test rows included). mlarena bench injects leaks from the Kapoor–Narayanan taxonomy (a published catalogue of leak types) into 5 clean datasets, and also runs 5 untouched control datasets.
VICTORY CONDITIONS
- find ≥ 11/12 leaks with ≤ 1 false alarm (planted leaks detected out of 12; false alarms on the 5 clean controls).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Data leakage
- Training data that contains information the model would not have in real use, which inflates test scores.
- Group leakage
- The same person, customer or object appearing in both train and test, so the model can memorise it instead of generalising.
- Post-outcome feature
- A column that was recorded after the outcome happened, so it partly gives away the answer.
- False alarm
- Your tool reporting a leak in a dataset that actually has none.
Why it rates ★★☆☆☆
Understanding leakage is what separates practitioners from tutorial-followers, and hiring managers notice it.