← All Base Form ML projects
BL-22NOT STARTEDBASE FORM · TRANSFORMERSMLPYTHON

Fast GPT

Make your tiny GPT generate faster with a KV cache and 8-bit weights, with a tokens-per-second dashboard.

EST.8 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
speed; output match; perplexity
GOAL · identical greedy output; ≥ 3× faster; ≤ 2% perplexity loss
your uncached fp32 model—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

This project takes the tiny GPT you trained in BL-21 and makes it generate faster, without changing what it says. That part of the work is called inference. Training sees whole sequences at once. Generation is different: it adds one token at a time, and the naive loop runs the full model over the whole prefix again for every new token.

Most of that work repeats. The keys and values for tokens 1 to t−1t-1 are the same at step tt as they were at step t−1t-1, because of the causal mask. A KV cache stores them. Each step then computes qq, kk and vv only for the new token and attends over the cached keys. The cost of a step goes from a full pass over tt tokens to one token's worth of work plus one row of attention. The cache costs memory instead:

bytes=2×layers×T×dmodel×bytes per value\text{bytes} = 2 \times \text{layers} \times T \times d_\text{model} \times \text{bytes per value}

for one sequence of length TT. The 2 is for keys and values.

The second idea is quantisation. A weight matrix stored as float32 uses 4 bytes per number. Symmetric int8 stores each row as integers q=round(w/s)q = \text{round}(w / s) in [−127,127][-127, 127] with one float scale s=max⁡∣w∣/127s = \max|w| / 127, and recovers w^=s q\hat w = s\,q. That is a quarter of the memory with a small rounding error, and perplexity tells you how much the error costs. In plain PyTorch on a CPU, dequantising on the fly may not be faster than fp32. The speed win here comes from the cache; the int8 win is memory and file size. BL-27 serves this cached model, so keep the generate function clean.

VISUALFig. 1
WITHOUT CACHE Step 5 recomputes everything K V tok 1 tok 2 tok 3 tok 4 tok 5 5 rows computed this step; t rows at step t WITH KV CACHE Step 5 computes one row K V tok 1 tok 2 tok 3 tok 4 tok 5 grey: read from the cache · yellow: computed now 1 row computed per step, whatever t is
At step 5, the uncached model recomputes keys and values for all five tokens. With a cache it computes one new row and reads the other four. Why it's here: Shows exactly which work the cache removes, so you know what objective 2 should change.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.

DATA · your BL-21 model

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. The KV-cached model gives identical greedy output on all 20 prompts and is at least 3× faster in tokens per second at full context length. Record it with `python tools/mlarena.py record BL-22 "<speedup>x, ppl +<percent>%"`.
  2. The int8 model's validation perplexity is no more than 2% above the fp32 model's.
  3. The int8 safetensors file is under a third the size of the same weights saved as fp32 safetensors, and the dashboard shows all variants.

Side quests

Try 4-bit weights with one scale per group of 32 values and see where perplexity breaks.
Read how the GGUF format used by llama.cpp lays out quantised blocks and compare it with your file.

Terms in plain words

KV cache
The keys and values of tokens already processed, kept in memory so each new token only computes its own.
Greedy decoding
Always pick the single most likely next token. It's deterministic, so two implementations can be compared token for token.
Quantisation
Storing weights as small integers plus a scale factor, such as int8 instead of float32, to cut memory four times.
Perplexity
The exponential of the average cross-entropy loss; roughly how many tokens the model is choosing between at each step.
safetensors
A simple file format that stores named tensors as a small header plus raw bytes, so weights load fast and safely.