← All Cracked Math projects
PM-W04NOT STARTEDWOWMATHPYTHONIMPACT ★★★★☆

LLM as compressor: GPT-2 plus your arithmetic coder vs cmix

Prediction is compression: drive a bit-exact coder with GPT-2 probabilities and see whether it beats the best hand-built compressor.

EST.55 h
REWARD+600 XP
AFTERDeletang et al. 2023 + 2

ML-ARENA BENCHMARKS

PROOF: FINDING
METRICINDUSTRYMINE
Bits per byte on the 10 MB prefix; total size including decompressor and weights
GOAL · Bit-exact decode on 2 machines; beat cmix's rate on the same prefix excluding weights; a published model-size-adjusted break-even analysis (how much data you must compress before GPT-2's weights pay for themselves), with a contamination note (GPT-2's WebText training data excluded Wikipedia, which matters for whether the model has already seen the test text)
cmix v21 on the same prefix · measured locally (LTCB full enwik8: cmix v21 14,623,723; nncp v3.2 14,915,298 bytes); Deletang et al. 2023, Language Modeling Is Compression—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Drive your I09 arithmetic coder with next-token probabilities from the A03 NumPy GPT-2, on a 10 MB enwik8 prefix split into parallel streams (independent chunks that can be coded at the same time). Solve determinism, so that decoding on a second machine reproduces encoding exactly: the decoder must compute the exact same probabilities, down to the last bit. Do this with quantised probabilities (rounded to fixed integer values) and fixed-order reductions (always adding numbers in the same order, because floating-point addition order changes the result). Add an online per-file mixer/SSE stage (secondary symbol estimation: a small model that corrects GPT-2's probabilities based on how well they have been doing on this file), fit by MLE while coding. Report both the raw rate and the rate including model size.

VICTORY CONDITIONS

  1. Bit-exact decode on 2 machines; beat cmix's rate on the same prefix excluding weights; a published model-size-adjusted break-even analysis (how much data you must compress before GPT-2's weights pay for themselves), with a contamination note (GPT-2's WebText training data excluded Wikipedia, which matters for whether the model has already seen the test text) (Bits per byte on the 10 MB prefix; total size including decompressor and weights).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

cmix
A top-ranked, very slow compressor that mixes the predictions of thousands of hand-built models.
Bits per byte
Compressed size in bits divided by the original size in bytes; 8 means no compression.
Determinism
Getting bit-for-bit identical results on every run and machine, which arithmetic decoding needs because one differing probability corrupts the rest of the file.
SSE (secondary symbol estimation)
A small adaptive stage that maps a model's predicted probability to a corrected one, based on how often such predictions were right.
Data contamination
When test data was part of a model's training set, which would make its results look better than they really are.

Why it rates ★★★★☆

Bit-exact neural compression with an honest size-adjusted analysis is publishable-quality work that touches LLM internals.