Brief
In BL-22 you made one generation fast. A service has a different problem: many people ask at once. Serving is the work of running a model for many clients and measuring it the way they experience it. Two numbers matter. Latency is how long one request takes; you report the median (p50) and the 95th percentile (p95), because averages hide the slow tail. Throughput is how many tokens the whole server produces per second.
A one-at-a-time server wastes most of the hardware. One sequence through a small GPT is a thin matrix multiply, and the CPU or GPU could do several sequences in nearly the same time. Dynamic batching uses that. Requests wait a few milliseconds in a queue, then go through the model together. Each request may wait a little longer to start, but the queue drains much faster, so under load p95 often gets better too.
The pieces are a web framework (FastAPI), an asyncio queue, a worker thread for the model so the event loop stays free, and a batched version of your BL-22 KV-cache generate. The cache now has a batch dimension, so its size is for a batch of . That number caps how big a batch you can take.
DATA · your BL-22 model
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- At the same request rate, the batching server delivers at least 3× the baseline's throughput with a p95 no more than 20% worse. Record it with `python tools/mlarena.py record BL-27 "<x>x tok/s, p95 <ms> ms"`.
- Every prompt gets the same greedy output batched as it does alone.
- The README has the throughput against p95 plot and names the max_batch and max_wait you would ship.
Side quests
Terms in plain words
- Latency
- The time from sending one request to getting its full answer back, as the client sees it.
- p50 / p95
- The median latency, and the latency that 95% of requests beat. p95 shows what your slow users feel.
- Throughput
- How much work the server finishes per second, counted here in generated tokens per second.
- Dynamic batching
- Holding requests for a few milliseconds so the server can run several of them through the model in one forward pass.