Brief
Load GPT-2 small weights (safetensors format) into a NumPy forward pass in the style of picoGPT (a minimal NumPy implementation of GPT-2). Compress every linear layer by replacing its weight matrix with a product of two thin matrices (a low-rank approximation), in three ways: (a) plain truncated SVD (keep only the top singular values); (b) activation-whitened SVD via a Cholesky of calibration activations, which is the SVD-LLM idea: measure the error by how the layer's real inputs are spread, so you discard directions the inputs rarely use; and (c) your I04 randomized SVD. Allocate per-layer ranks from singular-value energy (keep enough singular values to retain a chosen share of the sum of their squares). Then quantise the factors to int8 (8-bit integers) and measure how conditioning amplifies the error, since badly conditioned factors magnify rounding.
Approach
GPT-2 stores each linear layer as an input-by-output matrix and computes . Stack a layer's inputs from a calibration set into , one token per row. What you care about is the error on those inputs, not on all inputs:
So take the SVD of , keep the top singular values, and map back: . The output error is then exactly of the whitened matrix, which gives you both a free correctness check and a cheap per-layer score for rank allocation. Store the result as two thin factors, an input-by- matrix and a -by-output matrix. ASVD uses a diagonal scaling from per-channel activation size instead of the full Cholesky factor, which makes it a natural midpoint to compare.
Count before you compress. An matrix stored as rank- factors costs numbers, so it only saves anything when : 384 for a 768 × 768 projection, about 614 for the 768 × 3072 MLP layers. GPT-2 small has 124,439,808 parameters and about 84.9 million of them are linear-layer weights. A 20% cut of the whole model is a much deeper cut of the linear layers, so say which denominator you use.
Build the reference forward pass first and do not move on until the logits match. Then plain SVD on one layer, whitened SVD on the same layer with the error identity checked, then the whole model, then allocation and int8.
Where people get stuck
- Transposed whitening. With GPT-2's input-by-output layout the Cholesky factor multiplies from the left of . Most write-ups use the output-by-input layout and multiply from the right. Check the error identity on one layer before trusting either.
- Cholesky fails. for the 3072-wide MLP projection is singular unless you have well over 3072 calibration tokens, and GPT-2 has a few input channels with huge values. Add a small multiple of the mean diagonal before factorising, and report the amount.
- Leaking the test set. Calibration text must come from the training split. Calibrating on the WikiText-2 test text makes the method look better than it is.
- Perplexity that moves on its own. Window length and stride change GPT-2's perplexity by several points. Fix them once, and compare every method under the same setting.
- int8 on badly conditioned factors. can have a huge condition number, so the left factor may hold a few enormous entries. One scale per matrix then rounds everything else to zero. Use per-row or per-column scales, or fold into both factors to balance them.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- At 20% parameter reduction, whitened SVD raises WikiText-2 perplexity by at most half as much as plain SVD; a searched rank allocation beats uniform rank; the full Pareto curve is published. Record with `python tools/mlarena.py record PM-A03 "+<whitened> vs +<plain> ppl at 20%"`.
- The uncompressed NumPy model matches the reference logits to 1e-4, so every perplexity change comes from compression.
- The write-up states the parameter denominator (whole model or linear weights only) and the perplexity window and stride.
Side quests
Terms in plain words
- Perplexity
- The exponential of a language model's average loss per token; lower means the model predicts the text better.
- Truncated SVD
- Keeping only the top singular values and vectors of a matrix, which gives the best low-rank approximation in the usual norms.
- Activation whitening
- Transforming a layer's weights by the Cholesky factor of its input covariance, so the SVD minimises error on real inputs rather than on all directions equally.
- Pareto curve
- The set of best achievable trade-offs, here the lowest perplexity reachable at each parameter count.
- int8 quantisation
- Storing numbers as 8-bit integers with a scale factor, which saves memory but adds rounding error.
- Calibration set
- A small sample of real text run through the model only to measure each layer's input statistics. It must not overlap the evaluation text.
- Rank allocation
- How many singular values each layer keeps. Layers differ in how much they tolerate compression, so equal ranks are rarely best.
Why it rates ★★★☆☆
Hands-on LLM compression with real numbers and no framework, though the method is not new.