Brief
This project takes the tiny GPT you trained in BL-21 and makes it generate faster, without changing what it says. That part of the work is called inference. Training sees whole sequences at once. Generation is different: it adds one token at a time, and the naive loop runs the full model over the whole prefix again for every new token.
Most of that work repeats. The keys and values for tokens 1 to are the same at step as they were at step , because of the causal mask. A KV cache stores them. Each step then computes , and only for the new token and attends over the cached keys. The cost of a step goes from a full pass over tokens to one token's worth of work plus one row of attention. The cache costs memory instead:
for one sequence of length . The 2 is for keys and values.
The second idea is quantisation. A weight matrix stored as float32 uses 4 bytes per number. Symmetric int8 stores each row as integers in with one float scale , and recovers . That is a quarter of the memory with a small rounding error, and perplexity tells you how much the error costs. In plain PyTorch on a CPU, dequantising on the fly may not be faster than fp32. The speed win here comes from the cache; the int8 win is memory and file size. BL-27 serves this cached model, so keep the generate function clean.
DATA · your BL-21 model
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- The KV-cached model gives identical greedy output on all 20 prompts and is at least 3× faster in tokens per second at full context length. Record it with `python tools/mlarena.py record BL-22 "<speedup>x, ppl +<percent>%"`.
- The int8 model's validation perplexity is no more than 2% above the fp32 model's.
- The int8 safetensors file is under a third the size of the same weights saved as fp32 safetensors, and the dashboard shows all variants.
Side quests
Terms in plain words
- KV cache
- The keys and values of tokens already processed, kept in memory so each new token only computes its own.
- Greedy decoding
- Always pick the single most likely next token. It's deterministic, so two implementations can be compared token for token.
- Quantisation
- Storing weights as small integers plus a scale factor, such as int8 instead of float32, to cut memory four times.
- Perplexity
- The exponential of the average cross-entropy loss; roughly how many tokens the model is choosing between at each step.
- safetensors
- A simple file format that stores named tensors as a small header plus raw bytes, so weights load fast and safely.