Brief
In pure Python, implement the GPT-2 regex pre-tokenizer (a regular expression that first splits text into word-like chunks), byte-level mapping (treat text as raw UTF-8 bytes so any character can be encoded), and merge-rank BPE encode and decode. BPE (byte-pair encoding) repeatedly merges the adjacent pair of pieces that has the lowest rank in a learned merge table. Load the merge tables for gpt2 and o200k_base (GPT-4o's tokenizer) and fuzz against tiktoken (compare outputs on large amounts of varied input) on 50 MB of mixed text. Then train your own 32k-token BPE vocabulary on a code corpus and compare compression. minbpe already matches cl100k, so the twist here is o200k plus a throughput target.
VICTORY CONDITIONS
- 100% exact match; ≥ 25% of tiktoken's throughput; your own 32k vocab gets ≥ 5% more bytes/token than gpt2 on held-out code (token-ID exact-match rate for gpt2 and o200k_base on 50 MB; encoding speed in MB/s; bytes per token on held-out code (more bytes per token means better compression)).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- BPE (byte-pair encoding)
- A tokenizer that starts from single bytes and repeatedly merges the most useful adjacent pairs into new tokens.
- Pre-tokenizer
- A first pass, here a regex, that splits text into chunks so BPE merges never cross chunk boundaries.
- Merge rank
- The position of a pair in the learned merge list; lower-ranked pairs are merged first.
- Fuzzing
- Testing by feeding huge amounts of varied or strange input and checking that two implementations always agree.
- Bytes per token
- How many bytes of text each token covers on average; higher means the tokenizer compresses better.
Why it rates ★☆☆☆☆
minbpe made this a well-trodden path. The exact-match fuzzing is the only part that stands out.