← All Base Form ML projects
BL-23NOT STARTEDBASE FORM · TRANSFORMERSMLPYTHON

Faster training

Train a 10M-parameter model in under an hour with mixed precision, gradient accumulation and parallel data loading.

EST.8 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
time to target loss
GOAL · ≥ 1.5× faster, same loss
your fp32 baseline · measured—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In BL-21 you trained a small model and waited. This project is about waiting less. Training at scale means getting the most tokens through the model per second without changing what it learns, so the honest measure is not time per step but time until validation loss reaches a target you fix in advance.

Three changes do most of the work. Mixed precision runs matrix multiplies in 16-bit floats, which halves memory traffic and uses the fast tensor cores on most GPUs. bfloat16 has the same exponent range as float32, so it just works. float16 has a much smaller range, so tiny gradients underflow to zero unless you scale the loss up first. Gradient accumulation lets you use a big batch on a small machine: with kk micro-batches of size bb,

∇Lbatch=1k∑i=1k∇Li\nabla L_{\text{batch}} = \frac{1}{k} \sum_{i=1}^{k} \nabla L_i

which is the same gradient as one batch of size kbkb. The third change is parallel: while the model computes one batch, other workers prepare the next. Python threads share one interpreter lock, so heavy Python work goes in worker processes. Work that mostly waits on the disk or the GPU can use a thread, because NumPy and PyTorch release the lock inside many of their operations.

You need a model big enough for this to matter. A GPT with LL layers and width dd has about 12Ld212Ld^2 weights outside the embeddings, so 6 layers at width 384 gives roughly 10.6M. Mixed precision gains are much larger on a GPU. On a CPU, bfloat16 autocast still runs, but measure before you expect a win.

VISUALFig. 1
SERIAL main L C1 L C2 L C3 L C4 OVERLAPPED loader main L L L L C1 C2 C3 C4 done done TIME L: prepare a batch C: forward, backward, step
Serial loading leaves the model idle while each batch is prepared. With a background loader, the next batch is ready when the model needs it. Why it's here: Objective 3 is about closing these gaps; the picture shows where the lost time is.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.

DATA · a public text corpus

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. The fast run reaches the same target validation loss at least 1.5× sooner than your fp32 baseline. Record it with `python tools/mlarena.py record BL-23 "<speedup>x to loss <target>"`.
  2. The fast run's final validation loss is no worse than the baseline's after the same number of tokens.
  3. The 10M-parameter model reaches the target loss in under one hour of wall-clock time, and the accumulation test passes.

Side quests

Add torch.compile and measure what it adds on top of mixed precision.
If you have two GPUs, train with DistributedDataParallel and see how close you get to 2× tokens per second.

Terms in plain words

Mixed precision
Doing most of the maths in 16-bit floats (float16 or bfloat16) while keeping the master weights and some sums in float32.
Loss scaling
Multiplying the loss before backward so small float16 gradients don't round to zero, then dividing the gradients back before the step.
Gradient accumulation
Running several small batches and adding up their gradients before one optimiser step, to train with a batch too big for memory.
Data loader worker
A separate process that prepares the next batches while the main process trains, so the model never waits for data.
Time to target loss
Wall-clock time until validation loss first reaches a fixed value. It is fairer than comparing time per step.