← All Cracked ML projects
PL-I07NOT STARTEDINTERMEDIATEMLPYTHONIMPACT ★★☆☆☆

GPT-2 in NumPy that matches Hugging Face to 1e-4

picoGPT plus proof: logits equal to Hugging Face (HF) transformers, a KV cache, and decode speed that beats HF on CPU.

EST.14 h
REWARD+300 XP
AFTERJay Mody

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
max |logit diff| vs HF over 50 prompts; decode tokens/s at batch 1 (fp32, CPU); KV-cache speedup at 512 tokens
GOAL · max diff ≤ 1e-4; KV cache ≥ 10× faster than no cache at 512 tokens; decode ≥ 1.0× HF's CPU tok/s
Hugging Face transformers GPT2LMHeadModel on CPU · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Write your own safetensors reader (safetensors is the file format the weights ship in), load GPT-2 124M/355M, and implement the forward pass in NumPy: causal multi-head attention (each token attends only to earlier tokens), GELU activations and pre-LN (LayerNorm applied before each sub-block rather than after). Add a KV cache (store each layer's keys and values for past tokens, so each new token costs one step instead of re-running the whole sequence), temperature/top-k/top-p sampling, and batched prompts with padding masks (so shorter prompts in a batch ignore their padding). Tokenisation reuses PL-B07.

VICTORY CONDITIONS

  1. max diff ≤ 1e-4; KV cache ≥ 10× faster than no cache at 512 tokens; decode ≥ 1.0× HF's CPU tok/s (max |logit diff| vs HF over 50 prompts; decode tokens/s at batch 1 (fp32, CPU); KV-cache speedup at 512 tokens).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

picoGPT
A famous very short NumPy implementation of the GPT-2 forward pass.
safetensors
A simple, safe file format for storing model weights as named raw arrays with a JSON header.
KV cache
Saving the attention keys and values of earlier tokens so generating the next token does not recompute them.
Pre-LN
Placing LayerNorm at the start of each attention and MLP block, which is how GPT-2 is built.
Padding mask
A mask that stops attention from looking at filler tokens added to make prompts the same length.

Why it rates ★★☆☆☆

Everyone has seen picoGPT, a well-known tiny NumPy GPT-2. The exact-match test and beating HF's speed make yours credible, not novel.