← All Cracked ML projects
PL-A13NOT STARTEDADVANCEDMLPYTHONIMPACT ★★★☆☆

LLM serving engine: paged KV, continuous batching, int8

A vLLM in miniature, from a zero-copy weight loader to an OpenAI-compatible endpoint, measured against vLLM.

EST.40 h
REWARD+400 XP
AFTERKwon et al. + 1

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
output tokens/s; p50/p99 time-to-first-token (TTFT, the delay before the first token streams back) under load; int8 perplexity delta on WikiText-2
GOAL · ≥ 80% of vLLM's throughput at comparable p99 TTFT; int8 perplexity within +1% of bf16
vLLM on the same GPU · measured locally (nano-vllm README: 1434 vs vLLM 1362 tok/s on an RTX 4070 Laptop)—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Build an mmap-based zero-copy safetensors loader (map the weight file straight into memory instead of reading and copying it). Build a paged KV-cache block allocator with block tables: the KV cache is cut into fixed-size blocks, like pages of virtual memory, and each request keeps a table of which blocks it owns, so no memory is wasted on padding. Add prefix caching (requests that share a prompt prefix share its KV blocks), a continuous-batching scheduler with preemption (new requests join the running batch at every decode step, and running requests can be paused when memory runs out), CUDA-graph decode (record the GPU work of one decode step once and replay it, which cuts launch overhead), int8 weight-only quantisation and an OpenAI-compatible streaming HTTP server. Benchmark with the nano-vllm protocol (256 requests, random 100–1024 input/output lengths) on Qwen3-0.6B.

VICTORY CONDITIONS

  1. ≥ 80% of vLLM's throughput at comparable p99 TTFT; int8 perplexity within +1% of bf16 (output tokens/s; p50/p99 time-to-first-token (TTFT, the delay before the first token streams back) under load; int8 perplexity delta on WikiText-2).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Paged KV cache
Storing each request's attention keys and values in small fixed-size blocks that can sit anywhere in GPU memory.
Continuous batching
Adding and removing requests from the running batch at every generation step instead of waiting for a whole batch to finish.
Prefix caching
Reusing the stored keys and values for a prompt beginning that an earlier request already processed.
CUDA graph
A recorded sequence of GPU operations that can be replayed with a single launch, which removes per-kernel overhead.
TTFT
Time to first token: how long a user waits before the reply starts streaming.
Weight-only quantisation
Storing weights as int8 to save memory and bandwidth while still doing the maths in higher precision.

Why it rates ★★★☆☆

Inference infrastructure is where many ML jobs are, and a measured mini-vLLM is a strong portfolio piece even though nano-vllm exists.