← All Cracked ML projects
PL-W04NOT STARTEDWOWMLPYTHONIMPACT ★★★★★

Vesuvius Challenge: ink detection for a Progress Prize

Read text inside carbonised 2,000-year-old scrolls. Prizes are awarded monthly.

EST.60 h
REWARD+600 XP
AFTERVesuvius Challenge + 1

ML-ARENA BENCHMARKS

PROOF: UPSTREAM
METRICINDUSTRYMINE
late-submission score on the Kaggle Vesuvius Ink Detection private LB (leaderboard), using the competition metric; an accepted prize submission
GOAL · top-10% late-submission score, and an awarded Progress Prize or a merged open-source contribution
Kaggle Vesuvius Challenge Ink Detection (2023, 1,249 teams); scrollprize.org Progress Prizes ($250–$20,000 monthly, $20k best-of-month)—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Train 3D-CNN / 2.5D ink-detection models (networks that look at a stack of scan slices through the papyrus surface and predict where there is ink) on the public fragment and segment data. Fragments are detached pieces with known ink labels; segments are surfaces virtually unrolled from intact scrolls. Use careful leakage-free spatial validation (PL-B06 skills): hold out whole regions, so overlapping or neighbouring patches of the same area never end up in both train and validation. Score a late submission on the 2023 Kaggle ink-detection competition. Then turn the work into an open-source tool, or into an ink prediction on a new segment submitted for a monthly Progress Prize (or as a First Letters candidate).

Approach

The model looks at a thin stack of CT slices through the papyrus surface and paints a 2D ink mask. The competition gave three training fragments with 65 slices each, and the hidden test set is two fragments of about the size of one training fragment. The score is F0.5F_{0.5} for each test fragment, averaged:

F0.5=(1+0.52) p r0.52 p+r,p=TPTP+FP,r=TPTP+FN.F_{0.5} = \frac{(1 + 0.5^2)\, p\, r}{0.5^2\, p + r}, \qquad p = \frac{TP}{TP + FP}, \quad r = \frac{TP}{TP + FN}.

With β=0.5\beta = 0.5, a false ink pixel costs more than a missed one, because spurious strokes ruin letter shapes. Your threshold is therefore part of the model. Choose it on held-out regions, never on the pixels you trained on.

Build in this order: folds first, then a simple 2.5D baseline, then a 3D encoder. Ink signal is faint and labels come from infrared photos, so the honest validation score is noisy. Region hold-out is the only way to know whether a change helped. The archived private leaderboard (the copy in OpenAI's MLE-bench) has 1,250 entries. The winner scored 0.831, entry 125 scored 0.660 and the median was 0.418. Those are the numbers to place yourself against.

Compute is bounded by inference, not training. A Kaggle GPU notebook may run at most 9 hours with internet off, and offline pretraining is allowed if it is documented. Measure one fold's epoch time before you plan a sweep.

The prize work is on scroll data, not fragments. Data now lives in an open S3 bucket, mostly as OME-Zarr, with segments as tifxyz meshes. The community code is the ScrollPrize/villa monorepo, and the current public ink model is ink_canonical_2um on Hugging Face. Progress Prizes are judged monthly: a guaranteed $20,000 best submission of the month plus smaller awards. You must be registered on the Vesuvius Challenge Discord when you submit.

VISUALFig. 1
RANDOM PATCHESscore leaks validation patches (red) share pixels with training patches, so the model has seen them REGION HOLD-OUThonest score buffer ≥ 1 patch validationregion train only left of the buffer; rotate the held-out region across folds and fragments, and pick thresholds on it
Random overlapping patches put the same pixels in training and validation. Holding out a whole region with a buffer keeps the validation score honest. Why it's here: Objective 2 depends on this split, and a leaky split is the most common reason a Kaggle score falls far below cross-validation.

Where people get stuck

  • Leaky validation. Patches with stride smaller than their size overlap. A random patch split then tests the model on pixels it trained on, and the score looks far better than it is.
  • Threshold tuned on the wrong data. F0.5F_{0.5} is sensitive to the threshold. Tuning it on training pixels inflates precision. Tune per fold on held-out regions and check the chosen thresholds agree.
  • Mask edges. Pixels outside the papyrus mask are not part of the problem. Zero them in predictions, or tiles at the edge add false positives.
  • Slice choice. Ink sits near the papyrus surface, and the useful slices differ between fragments. Plot mean intensity against slice index per fragment before fixing a window.
  • Fragment success that doesn't transfer. Scroll segments differ from fragments in resolution and surface quality. A fragment gain is a hypothesis about scrolls until you test it on a segment.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

Objectives

0 / 8 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Late-submission F0.5 of at least 0.660 on the private leaderboard, the top 10% of the 1,250 archived entries, recorded with `python tools/mlarena.py record PL-W04 "<private F0.5>"`.
  2. An awarded Progress Prize, or a merged pull request in ScrollPrize/villa.
  3. A documented spatial validation protocol, with the gap between cross-validation and the private score reported.

Side quests

Measure how well fragment cross-validation predicts ink quality on scroll segments, using segments where text is already known. That is one of the open problems the Challenge lists.
Turn your model's probabilities into an uncertainty map that a papyrologist can use to decide where to look closer.

Terms in plain words

Vesuvius Challenge
An open effort to read Herculaneum scrolls, carbonised by Vesuvius in AD 79, from X-ray CT scans.
2.5D CNN
A CNN that treats a few stacked scan slices as input channels, a cheaper middle ground between 2D and full 3D.
Fragment vs segment
A fragment is a loose piece with visible, labelled ink; a segment is a surface digitally peeled from an unopened scroll.
Spatial validation
Splitting data by location so the validation area is truly unseen, instead of random patches that overlap training patches.
Progress Prize
A monthly award from the Vesuvius Challenge for useful tools or discoveries.
F0.5 score
A mix of precision and recall that weights precision more, so false ink costs more than missed ink.
Run-length encoding
Storing a binary mask as pairs of start position and run length, which is how Kaggle expects the predictions.

Why it rates ★★★★★

A prize-winning contribution to a famous science effort is a standout story for any hiring manager.