Brief
Drive your I09 arithmetic coder with next-token probabilities from the A03 NumPy GPT-2, on a 10 MB enwik8 prefix split into parallel streams (independent chunks that can be coded at the same time). Solve determinism, so that decoding on a second machine reproduces encoding exactly: the decoder must compute the exact same probabilities, down to the last bit. Do this with quantised probabilities (rounded to fixed integer values) and fixed-order reductions (always adding numbers in the same order, because floating-point addition order changes the result). Add an online per-file mixer/SSE stage (secondary symbol estimation: a small model that corrects GPT-2's probabilities based on how well they have been doing on this file), fit by MLE while coding. Report both the raw rate and the rate including model size.
VICTORY CONDITIONS
- Bit-exact decode on 2 machines; beat cmix's rate on the same prefix excluding weights; a published model-size-adjusted break-even analysis (how much data you must compress before GPT-2's weights pay for themselves), with a contamination note (GPT-2's WebText training data excluded Wikipedia, which matters for whether the model has already seen the test text) (Bits per byte on the 10 MB prefix; total size including decompressor and weights).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- cmix
- A top-ranked, very slow compressor that mixes the predictions of thousands of hand-built models.
- Bits per byte
- Compressed size in bits divided by the original size in bytes; 8 means no compression.
- Determinism
- Getting bit-for-bit identical results on every run and machine, which arithmetic decoding needs because one differing probability corrupts the rest of the file.
- SSE (secondary symbol estimation)
- A small adaptive stage that maps a model's predicted probability to a corrected one, based on how often such predictions were right.
- Data contamination
- When test data was part of a model's training set, which would make its results look better than they really are.
Why it rates ★★★★☆
Bit-exact neural compression with an honest size-adjusted analysis is publishable-quality work that touches LLM internals.