← All Cracked ML projects
PL-A10NOT STARTEDADVANCEDMLPYTHONIMPACT ★★★☆☆

GPT-2 124M trainer with your own data-parallel + ZeRO-1

No DDP wrapper: bucketed all-reduce that overlaps with the backward pass, sharded optimiser state, and a FineWeb val loss of 3.28.

EST.35 h
REWARD+400 XP
AFTERKarpathy + 1

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
FineWeb validation loss vs tokens seen; tokens/s scaling efficiency from 1→N GPUs; optimiser memory per GPU
GOAL · val loss ≤ 3.28 within 10B tokens; scaling efficiency ≥ 90% of torch DDP's; ZeRO-1 cuts optimiser memory per GPU by ≈ 1/N (each GPU holds about 1/N of the full optimiser state)
torch DDP (measured locally); llm.c GPT-2 124M reproduction (90 min on 8×A100, ~$20)—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Write a GPT-2 124M training stack on torch.distributed primitives only. It needs gradient bucketing with all-reduce overlapped with backward via hooks: group gradients into buckets and, as soon as a bucket is complete during the backward pass, start averaging it across GPUs (all-reduce) so communication hides behind computation. It also needs ZeRO-1 optimiser-state sharding (each GPU stores Adam's moment estimates for only 1/N of the parameters), bf16 autocast (run most of the maths in 16-bit bfloat16), gradient accumulation (sum gradients over several micro-batches before each optimiser step), a cosine/WSD schedule (WSD: warmup, a long stable phase, then a decay), checkpoint resume and a deterministic FineWeb data loader. Measure scaling from 1 to N GPUs against torch DDP.

VICTORY CONDITIONS

  1. val loss ≤ 3.28 within 10B tokens; scaling efficiency ≥ 90% of torch DDP's; ZeRO-1 cuts optimiser memory per GPU by ≈ 1/N (each GPU holds about 1/N of the full optimiser state) (FineWeb validation loss vs tokens seen; tokens/s scaling efficiency from 1→N GPUs; optimiser memory per GPU).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

All-reduce
A collective operation where every GPU ends up with the sum (or average) of all GPUs' tensors.
Gradient bucketing
Grouping many small gradients into larger buffers so each all-reduce call is efficient and can start early.
ZeRO-1
Splitting the optimiser's state across GPUs so each one stores and updates only its share.
bf16
A 16-bit float format with the same range as fp32 but less precision, used to train faster with less memory.
WSD schedule
A learning-rate schedule with warmup, a long constant phase, and a short decay at the end.
DDP
PyTorch's DistributedDataParallel wrapper, which does gradient syncing for you; here it is the baseline you must match.

Why it rates ★★★☆☆

Distributed-training plumbing written by hand is a strong signal for pretraining and infrastructure teams.