Brief
Write a GPT-2 124M training stack on torch.distributed primitives only. It needs gradient bucketing with all-reduce overlapped with backward via hooks: group gradients into buckets and, as soon as a bucket is complete during the backward pass, start averaging it across GPUs (all-reduce) so communication hides behind computation. It also needs ZeRO-1 optimiser-state sharding (each GPU stores Adam's moment estimates for only 1/N of the parameters), bf16 autocast (run most of the maths in 16-bit bfloat16), gradient accumulation (sum gradients over several micro-batches before each optimiser step), a cosine/WSD schedule (WSD: warmup, a long stable phase, then a decay), checkpoint resume and a deterministic FineWeb data loader. Measure scaling from 1 to N GPUs against torch DDP.
VICTORY CONDITIONS
- val loss ≤ 3.28 within 10B tokens; scaling efficiency ≥ 90% of torch DDP's; ZeRO-1 cuts optimiser memory per GPU by ≈ 1/N (each GPU holds about 1/N of the full optimiser state) (FineWeb validation loss vs tokens seen; tokens/s scaling efficiency from 1→N GPUs; optimiser memory per GPU).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- All-reduce
- A collective operation where every GPU ends up with the sum (or average) of all GPUs' tensors.
- Gradient bucketing
- Grouping many small gradients into larger buffers so each all-reduce call is efficient and can start early.
- ZeRO-1
- Splitting the optimiser's state across GPUs so each one stores and updates only its share.
- bf16
- A 16-bit float format with the same range as fp32 but less precision, used to train faster with less memory.
- WSD schedule
- A learning-rate schedule with warmup, a long constant phase, and a short decay at the end.
- DDP
- PyTorch's DistributedDataParallel wrapper, which does gradient syncing for you; here it is the baseline you must match.
Why it rates ★★★☆☆
Distributed-training plumbing written by hand is a strong signal for pretraining and infrastructure teams.