Brief
On your PL-A10 trainer, enter the BabyLM Strict-Small track: 10M words of training text, ≤ 10 epochs, and intermediate checkpoints are required. Test one clear hypothesis with ablations. Examples: curriculum by surprisal (show the model less surprising text first, with surprise scored by PL-B08's n-gram model), a tokenizer from PL-B07, or a latent-recurrence architecture (a model that runs the same layers several times over its hidden state). Score with the official babylm-eval pipeline.
Approach
The benchmark is a fixed budget, so the question is what you do inside it. Strict-Small allows 10M words of data and 100M words of exposure, counting repeats. Words are whitespace-separated tokens. You must save checkpoints every 1M words up to 10M, then every 10M up to 100M: 19 in all. The figure shows why that matters for a curriculum. Ten of the 19 checkpoints fall inside the first epoch, which is exactly where ordering the data changes anything.
Compute is not the constraint. With , a GPT-2 Small sized model () that reads 100M words at roughly 1.3 tokens per word sees tokens, so FLOPs. At a sustained 50 TFLOP/s that is about half an hour on one GPU. Measure your own tokens per word, since it depends on the tokenizer. The cheap runs are what make seeds and ablations affordable.
For a surprisal curriculum, score each training unit by its mean surprisal under the n-gram model:
Sort ascending and let a pacing function decide how much of the sorted data is available at each step. After the first epoch you can keep the order, reshuffle or reverse it. That choice is part of the hypothesis. The tasks are mostly zero-shot minimal-pair comparisons (BLiMP, BLiMP Supplement, EWoK, Entity Tracking, COMPS, GlobalPIQA), plus reading-time and age-of-acquisition fits and (Super)GLUE fine-tuning. A change that helps grammar can hurt fine-tuning, so read the per-task table, not only the aggregate.
Baselines in 2026 were GPT-BERT, a SimPO-based model from the previous Interaction track, and GPT-2 Small. The 2026 cycle is closed: direct submissions were due on July 20, 2026, and the workshop runs October 24 to 29, 2026 at EMNLP. Treat this as preparation for the next call, and recheck every rule when it appears.
Where people get stuck
- Counting tokens instead of words. The limits are in whitespace-separated words. A BPE tokenizer gives more tokens than words, so a token-based counter saves every checkpoint early and stops short of the budget you are allowed.
- Hidden exposure. Any model trained on language counts its training text toward the budget, and that includes a tokenizer or an n-gram scorer. Read the rule and write down how you count it.
- One seed. Differences of a point on the aggregate are often within seed noise at this scale. Three seeds per arm is the minimum, and the comparison must use the same compute.
- Incomplete submissions. Missing tasks are scored as 0 and still count in the average. Run the whole pipeline on your baseline first so the final run has no surprises.
- Confounded curricula. Sorting by surprisal also sorts by length and source. Include an arm that sorts by length alone, or the paper can't say which one helped.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Aggregate score above the official Strict-Small baseline on the BabyLM leaderboard, recorded with `python tools/mlarena.py record PL-W03 "<yours> vs <baseline>"`.
- The entry follows the rules: 10M words or fewer, no more than 100M words of exposure, and all 19 checkpoints public.
- An ablation with at least 3 seeds per arm that shows whether the effect is larger than seed noise, written up as a paper ready for the next edition.
Side quests
Terms in plain words
- BabyLM
- A yearly challenge to train language models on roughly the amount of text a child hears, with an academic workshop attached.
- Strict-Small
- The BabyLM track limited to 10M words of the provided data and nothing else.
- Sample efficiency
- How much a model learns per word of training data.
- Curriculum learning
- Ordering training data from easy to hard instead of shuffling it randomly.
- Surprisal
- The negative log-probability of text under a model; high surprisal means the text is unexpected.
- Ablation
- Turning off one change at a time to show which part caused the improvement.
- Exposure budget
- The total number of words a model may read during training, counting repeats; for Strict-Small it is 100M, usually 10 passes over 10M words.
- Pacing function
- In a curriculum, the rule that decides how much of the sorted data is available at each point in training.
- Paired bootstrap
- Resampling the same test items for both models many times to get an interval for the difference in their scores.
Why it rates ★★★★☆
A peer-reviewed workshop paper with a clean hypothesis is a real research credential.