← All Cracked ML projects
PL-B06NOT STARTEDBEGINNERMLPYTHONIMPACT ★★☆☆☆

Leak hunter: a leakage-audit CLI

The bench plants 12 leaks in 5 datasets. Your tool has to find them before they fool a model.

EST.7 h
REWARD+200 XP
AFTERKapoor & Narayanan

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
planted leaks detected out of 12; false alarms on the 5 clean controls
GOAL · find ≥ 11/12 leaks with ≤ 1 false alarm
Kapoor & Narayanan leakage taxonomy (Patterns 2023) · measured locally by mlarena bench—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Write a CLI that takes a tabular dataset, a target column and a split spec (which rows go to train and which to test), then flags data leaks. A leak is information that would not be available at prediction time but sneaks into training and makes the model look better than it is. Your tool must catch: suspiciously predictive single features; post-outcome timestamps (features recorded after the event you are predicting); exact and near-duplicate rows across splits; group leakage (the same entity, such as one patient or one customer, in both train and test); and preprocessing fitted before the split (for example, a scaler fitted on all rows, test rows included). mlarena bench injects leaks from the Kapoor–Narayanan taxonomy (a published catalogue of leak types) into 5 clean datasets, and also runs 5 untouched control datasets.

VICTORY CONDITIONS

  1. find ≥ 11/12 leaks with ≤ 1 false alarm (planted leaks detected out of 12; false alarms on the 5 clean controls).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Data leakage
Training data that contains information the model would not have in real use, which inflates test scores.
Group leakage
The same person, customer or object appearing in both train and test, so the model can memorise it instead of generalising.
Post-outcome feature
A column that was recorded after the outcome happened, so it partly gives away the answer.
False alarm
Your tool reporting a leak in a dataset that actually has none.

Why it rates ★★☆☆☆

Understanding leakage is what separates practitioners from tutorial-followers, and hiring managers notice it.