Brief
Build an mmap-based zero-copy safetensors loader (map the weight file straight into memory instead of reading and copying it). Build a paged KV-cache block allocator with block tables: the KV cache is cut into fixed-size blocks, like pages of virtual memory, and each request keeps a table of which blocks it owns, so no memory is wasted on padding. Add prefix caching (requests that share a prompt prefix share its KV blocks), a continuous-batching scheduler with preemption (new requests join the running batch at every decode step, and running requests can be paused when memory runs out), CUDA-graph decode (record the GPU work of one decode step once and replay it, which cuts launch overhead), int8 weight-only quantisation and an OpenAI-compatible streaming HTTP server. Benchmark with the nano-vllm protocol (256 requests, random 100–1024 input/output lengths) on Qwen3-0.6B.
VICTORY CONDITIONS
- ≥ 80% of vLLM's throughput at comparable p99 TTFT; int8 perplexity within +1% of bf16 (output tokens/s; p50/p99 time-to-first-token (TTFT, the delay before the first token streams back) under load; int8 perplexity delta on WikiText-2).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Paged KV cache
- Storing each request's attention keys and values in small fixed-size blocks that can sit anywhere in GPU memory.
- Continuous batching
- Adding and removing requests from the running batch at every generation step instead of waiting for a whole batch to finish.
- Prefix caching
- Reusing the stored keys and values for a prompt beginning that an earlier request already processed.
- CUDA graph
- A recorded sequence of GPU operations that can be replayed with a single launch, which removes per-kernel overhead.
- TTFT
- Time to first token: how long a user waits before the reply starts streaming.
- Weight-only quantisation
- Storing weights as int8 to save memory and bandwidth while still doing the maths in higher precision.
Why it rates ★★★☆☆
Inference infrastructure is where many ML jobs are, and a measured mini-vLLM is a strong portfolio piece even though nano-vllm exists.