← All Cracked Math projects
PM-A03NOT STARTEDADVANCEDMATHPYTHONIMPACT ★★★☆☆

Compress GPT-2 with activation-aware SVD, in pure NumPy

Plain SVD wrecks perplexity; whitening by the activations fixes most of it. Measure the whole Pareto curve of size against quality.

EST.25 h
REWARD+400 XP
AFTERJay Mody + 2

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
WikiText-2 perplexity (how surprised the model is by real text; lower is better) at 20% and 40% parameter reduction
GOAL · Whitened SVD's perplexity increase <= 50% of plain SVD's at 20% reduction; a searched rank allocation beats uniform rank; the full perplexity-vs-parameters Pareto curve is published
SVD-LLM (arXiv 2403.07378) and ASVD (arXiv 2312.05821) methods; GPT-2 numbers measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Load GPT-2 small weights (safetensors format) into a NumPy forward pass in the style of picoGPT (a minimal NumPy implementation of GPT-2). Compress every linear layer by replacing its weight matrix with a product of two thin matrices (a low-rank approximation), in three ways: (a) plain truncated SVD (keep only the top singular values); (b) activation-whitened SVD via a Cholesky of calibration activations, which is the SVD-LLM idea: measure the error by how the layer's real inputs are spread, so you discard directions the inputs rarely use; and (c) your I04 randomized SVD. Allocate per-layer ranks from singular-value energy (keep enough singular values to retain a chosen share of the sum of their squares). Then quantise the factors to int8 (8-bit integers) and measure how conditioning amplifies the error, since badly conditioned factors magnify rounding.

Approach

GPT-2 stores each linear layer as an input-by-output matrix and computes y=xW+by = xW + b. Stack a layer's inputs from a calibration set into XX, one token per row. What you care about is the error on those inputs, not on all inputs:

∥XW−XWk∥F2=∥L⊤(W−Wk)∥F2,X⊤X=LL⊤.\|XW - XW_k\|_F^2 = \|L^\top (W - W_k)\|_F^2, \qquad X^\top X = L L^\top .

So take the SVD of L⊤WL^\top W, keep the top kk singular values, and map back: Wk=L−⊤ [L⊤W]kW_k = L^{-\top}\,[L^\top W]_k. The output error is then exactly ∑i>kσi2\sqrt{\sum_{i > k} \sigma_i^2} of the whitened matrix, which gives you both a free correctness check and a cheap per-layer score for rank allocation. Store the result as two thin factors, an input-by-kk matrix and a kk-by-output matrix. ASVD uses a diagonal scaling from per-channel activation size instead of the full Cholesky factor, which makes it a natural midpoint to compare.

Count before you compress. An m×nm \times n matrix stored as rank-kk factors costs k(m+n)k(m + n) numbers, so it only saves anything when k<mn/(m+n)k < mn/(m + n): 384 for a 768 × 768 projection, about 614 for the 768 × 3072 MLP layers. GPT-2 small has 124,439,808 parameters and about 84.9 million of them are linear-layer weights. A 20% cut of the whole model is a much deeper cut of the linear layers, so say which denominator you use.

Build the reference forward pass first and do not move on until the logits match. Then plain SVD on one layer, whitened SVD on the same layer with the error identity checked, then the whole model, then allocation and int8.

VISUALFig. 1
above rank 128 the two factors cost more than W 0 0.25 0.5 0.75 1 0 64 128 192 256 plain SVD 0.43 whitened 0.06 rank 102 = 20% fewer params kept rank k (synthetic 256 × 256 layer) relative output error ‖WX − W_k X‖ / ‖WX‖
A synthetic 256 × 256 layer whose inputs use only a few directions. Plain SVD keeps directions the inputs rarely visit; whitened SVD keeps what matters on real inputs. Measured with NumPy on random data, not GPT-2. Why it's here: This is the whole idea of the project in one plot. Your GPT-2 layers should show the same gap, smaller or larger.

Where people get stuck

  • Transposed whitening. With GPT-2's input-by-output layout the Cholesky factor multiplies from the left of WW. Most write-ups use the output-by-input layout and multiply from the right. Check the error identity on one layer before trusting either.
  • Cholesky fails. X⊤XX^\top X for the 3072-wide MLP projection is singular unless you have well over 3072 calibration tokens, and GPT-2 has a few input channels with huge values. Add a small multiple of the mean diagonal before factorising, and report the amount.
  • Leaking the test set. Calibration text must come from the training split. Calibrating on the WikiText-2 test text makes the method look better than it is.
  • Perplexity that moves on its own. Window length and stride change GPT-2's perplexity by several points. Fix them once, and compare every method under the same setting.
  • int8 on badly conditioned factors. L−⊤L^{-\top} can have a huge condition number, so the left factor may hold a few enormous entries. One scale per matrix then rounds everything else to zero. Use per-row or per-column scales, or fold Σk1/2\Sigma_k^{1/2} into both factors to balance them.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

Objectives

0 / 7 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. At 20% parameter reduction, whitened SVD raises WikiText-2 perplexity by at most half as much as plain SVD; a searched rank allocation beats uniform rank; the full Pareto curve is published. Record with `python tools/mlarena.py record PM-A03 "+<whitened> vs +<plain> ppl at 20%"`.
  2. The uncompressed NumPy model matches the reference logits to 1e-4, so every perplexity change comes from compression.
  3. The write-up states the parameter denominator (whole model or linear weights only) and the perplexity window and stride.

Side quests

Compress the tied token-embedding matrix as well and see which tokens suffer most.
Repeat on GPT-2 medium and check whether whitening's advantage grows with model size.

Terms in plain words

Perplexity
The exponential of a language model's average loss per token; lower means the model predicts the text better.
Truncated SVD
Keeping only the top singular values and vectors of a matrix, which gives the best low-rank approximation in the usual norms.
Activation whitening
Transforming a layer's weights by the Cholesky factor of its input covariance, so the SVD minimises error on real inputs rather than on all directions equally.
Pareto curve
The set of best achievable trade-offs, here the lowest perplexity reachable at each parameter count.
int8 quantisation
Storing numbers as 8-bit integers with a scale factor, which saves memory but adds rounding error.
Calibration set
A small sample of real text run through the model only to measure each layer's input statistics. It must not overlap the evaluation text.
Rank allocation
How many singular values each layer keeps. Layers differ in how much they tolerate compression, so equal ranks are rarely best.

Why it rates ★★★☆☆

Hands-on LLM compression with real numbers and no framework, though the method is not new.