Brief
Implement five membership scores from per-token log-probs on Pythia models, whose training data (the Pile) is public. Each score guesses whether a given text was in the training data. Perplexity: training texts look less surprising to the model. zlib: perplexity compared with how compressible the text is, so simple, repetitive text is not mistaken for memorised text. Reference-model: compare against a second model that did not see the text. Min-K%: average the log-probs of only the k% least likely tokens. Min-K%++: the same, but each token's log-prob is first normalised by the mean and spread of the model's predicted distribution at that position. Reproduce the WikiMIA AUROCs. Then build an n-gram overlap index over a Pile shard, using either a suffix array (a sorted list of every suffix of the text, for fast exact substring search) or a Bloom filter (a compact bit array that answers 'definitely not present' or 'probably present'). Publish contamination rates for one popular benchmark, cross-checked against the membership scores.
Approach
Every score here comes from the same quantity: the log-prob of each real token. Perplexity averages it. Min-K% averages only the lowest , because members rarely contain tokens the model found shocking. Min-K%++ first asks how surprising the token is compared with what the model expected at that position:
and then averages the lowest of the . Both and are sums over the full vocabulary, so you get them from the same logits at no extra model cost. Compute them in fp32 from log_softmax, not from fp16 probabilities.
Build in this order: one function that returns per-token ; a cached table of those for every WikiMIA text; then each score is a few lines of NumPy on the cache. AUROC only depends on ranking, so you can check it against sklearn.metrics.roc_auc_score. The paper's Pythia results (Table 5) are for 2.8B, 6.9B and 12B, with Pythia-70M as the reference model. 2.8B and 6.9B run in fp16 on one 24 GB GPU; WikiMIA has only 1,650 texts, so each model takes minutes. 12B needs a rented 40 or 80 GB card for an hour or two.
The contamination half is a separate engineering problem. The full original Pile is hard to get as plain text today, so check what you can download first. Two public routes: monology/pile-uncopyrighted (text, with the copyrighted subsets removed, so a miss there is weaker evidence) or the tokenized EleutherAI/pile-standard-pythia-preshuffled (602 GB, Pythia's exact training order). Start with a shard of a few GB. A suffix array over bytes takes about bytes of index with 64-bit offsets and answers any substring query by binary search; a Bloom filter is far smaller but only answers fixed-length n-grams and has a false-positive rate for items, bits and hashes.
Where people get stuck
- Off-by-one in log-prob gathering. Logits at position predict token . Shift the targets, drop the first token, and test on a two-token string by hand.
- Tokenizer drift. WikiMIA texts are whole strings. Tokenize without truncation, and do not let a BOS token appear for one model and not another.
- Picking k on the test set. k changes AUROC by several points. Choose it on a held-out half or report the full k curve.
- Text normalisation in the index. Lowercasing, whitespace and Unicode forms decide whether a benchmark item "matches". Normalise the corpus and the queries identically and say how.
- Reading overlap as proof. A 13-gram hit may be a common phrase or a template. Hand-check samples before you publish a rate.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Min-K%++ AUROC within 2 points of the paper's WikiMIA numbers (Table 5, original setting) at lengths 32, 64 and 128 for at least two Pythia sizes. Record it with `python tools/mlarena.py record PL-A01 "<mean |gap| in pts>"`.
- A written contamination report for one benchmark, giving the overlap rate per split with the n-gram length and matching rules stated.
- The report includes a cross-check between overlap and membership scores, with a confidence interval, even if the answer is "no detectable relation".
Side quests
Terms in plain words
- Membership inference
- Guessing, from a model's outputs alone, whether a particular text was part of its training data.
- Contamination
- Benchmark questions or answers appearing in a model's training data, which makes its benchmark scores look better than they are.
- Min-K%++
- A membership score that looks at a text's least likely tokens, after normalising each token's log-prob by what the model expected at that position.
- AUROC
- Area under the ROC curve: the chance that a random member gets a higher score than a random non-member.
- Suffix array
- A sorted index of every suffix of a large text, which lets you check whether any string occurs in it very quickly.
- Bloom filter
- A small bit array with several hash functions that can tell you an item is definitely absent or probably present.
Why it rates ★★★☆☆
Contamination (benchmark test data leaking into training data) is an active, valued eval-integrity problem. An original audit is a real contribution at the level of a solid blog post.