Brief
In BL-21 you trained a small model and waited. This project is about waiting less. Training at scale means getting the most tokens through the model per second without changing what it learns, so the honest measure is not time per step but time until validation loss reaches a target you fix in advance.
Three changes do most of the work. Mixed precision runs matrix multiplies in 16-bit floats, which halves memory traffic and uses the fast tensor cores on most GPUs. bfloat16 has the same exponent range as float32, so it just works. float16 has a much smaller range, so tiny gradients underflow to zero unless you scale the loss up first. Gradient accumulation lets you use a big batch on a small machine: with micro-batches of size ,
which is the same gradient as one batch of size . The third change is parallel: while the model computes one batch, other workers prepare the next. Python threads share one interpreter lock, so heavy Python work goes in worker processes. Work that mostly waits on the disk or the GPU can use a thread, because NumPy and PyTorch release the lock inside many of their operations.
You need a model big enough for this to matter. A GPT with layers and width has about weights outside the embeddings, so 6 layers at width 384 gives roughly 10.6M. Mixed precision gains are much larger on a GPU. On a CPU, bfloat16 autocast still runs, but measure before you expect a win.
DATA · a public text corpus
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- The fast run reaches the same target validation loss at least 1.5× sooner than your fp32 baseline. Record it with `python tools/mlarena.py record BL-23 "<speedup>x to loss <target>"`.
- The fast run's final validation loss is no worse than the baseline's after the same number of tokens.
- The 10M-parameter model reaches the target loss in under one hour of wall-clock time, and the accumulation test passes.
Side quests
Terms in plain words
- Mixed precision
- Doing most of the maths in 16-bit floats (float16 or bfloat16) while keeping the master weights and some sums in float32.
- Loss scaling
- Multiplying the loss before backward so small float16 gradients don't round to zero, then dividing the gradients back before the step.
- Gradient accumulation
- Running several small batches and adding up their gradients before one optimiser step, to train with a batch too big for memory.
- Data loader worker
- A separate process that prepares the next batches while the main process trains, so the model never waits for data.
- Time to target loss
- Wall-clock time until validation loss first reaches a fixed value. It is fairer than comparing time per step.