← All Cracked ML projects
PL-B07NOT STARTEDBEGINNERMLPYTHONIMPACT ★☆☆☆☆

Byte-exact BPE: match tiktoken, then beat it on code

Your tokenizer must reproduce GPT-2 and GPT-4o token IDs bit for bit, including on emoji, CJK (Chinese, Japanese, Korean) text and code.

EST.7 h
REWARD+200 XP
AFTERKarpathy + 1

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
token-ID exact-match rate for gpt2 and o200k_base on 50 MB; encoding speed in MB/s; bytes per token on held-out code (more bytes per token means better compression)
GOAL · 100% exact match; ≥ 25% of tiktoken's throughput; your own 32k vocab gets ≥ 5% more bytes/token than gpt2 on held-out code
tiktoken (claims to be 3–6× faster than comparable open-source tokenisers) · throughput measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In pure Python, implement the GPT-2 regex pre-tokenizer (a regular expression that first splits text into word-like chunks), byte-level mapping (treat text as raw UTF-8 bytes so any character can be encoded), and merge-rank BPE encode and decode. BPE (byte-pair encoding) repeatedly merges the adjacent pair of pieces that has the lowest rank in a learned merge table. Load the merge tables for gpt2 and o200k_base (GPT-4o's tokenizer) and fuzz against tiktoken (compare outputs on large amounts of varied input) on 50 MB of mixed text. Then train your own 32k-token BPE vocabulary on a code corpus and compare compression. minbpe already matches cl100k, so the twist here is o200k plus a throughput target.

VICTORY CONDITIONS

  1. 100% exact match; ≥ 25% of tiktoken's throughput; your own 32k vocab gets ≥ 5% more bytes/token than gpt2 on held-out code (token-ID exact-match rate for gpt2 and o200k_base on 50 MB; encoding speed in MB/s; bytes per token on held-out code (more bytes per token means better compression)).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

BPE (byte-pair encoding)
A tokenizer that starts from single bytes and repeatedly merges the most useful adjacent pairs into new tokens.
Pre-tokenizer
A first pass, here a regex, that splits text into chunks so BPE merges never cross chunk boundaries.
Merge rank
The position of a pair in the learned merge list; lower-ranked pairs are merged first.
Fuzzing
Testing by feeding huge amounts of varied or strange input and checking that two implementations always agree.
Bytes per token
How many bytes of text each token covers on average; higher means the tokenizer compresses better.

Why it rates ★☆☆☆☆

minbpe made this a well-trodden path. The exact-match fuzzing is the only part that stands out.