← All Base Form ML projects
BL-27NOT STARTEDBASE FORM · TRANSFORMERSMLPYTHON

Serve your GPT

Put your tiny GPT behind a local API that batches requests, and measure latency like a real service.

EST.7 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
p50/p95 latency; throughput
GOAL · ≥ 3× throughput at similar p95
one-request-at-a-time server—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In BL-22 you made one generation fast. A service has a different problem: many people ask at once. Serving is the work of running a model for many clients and measuring it the way they experience it. Two numbers matter. Latency is how long one request takes; you report the median (p50) and the 95th percentile (p95), because averages hide the slow tail. Throughput is how many tokens the whole server produces per second.

A one-at-a-time server wastes most of the hardware. One sequence through a small GPT is a thin matrix multiply, and the CPU or GPU could do several sequences in nearly the same time. Dynamic batching uses that. Requests wait a few milliseconds in a queue, then go through the model together. Each request may wait a little longer to start, but the queue drains much faster, so under load p95 often gets better too.

The pieces are a web framework (FastAPI), an asyncio queue, a worker thread for the model so the event loop stays free, and a batched version of your BL-22 KV-cache generate. The cache now has a batch dimension, so its size is 2×layers×B×T×dmodel×bytes per value2 \times \text{layers} \times B \times T \times d_\text{model} \times \text{bytes per value} for a batch of BB. That number caps how big a batch you can take.

VISUALFig. 1
arrivals ONE ATA TIME r1 r2 r3 r4 r5 r6 r7 r8 done BATCHED r1 r2 r3 r4 r5 r6 r7 r8 done TIME request arrives collector waits up to max_wait one batched pass
Eight requests arrive. One at a time, they queue behind each other. Batched, the collector groups whatever arrived within max_wait and runs it as one pass. Why it's here: Shows why throughput rises and where the extra wait in objective 4 comes from.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.

DATA · your BL-22 model

Objectives

0 / 5 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. At the same request rate, the batching server delivers at least 3× the baseline's throughput with a p95 no more than 20% worse. Record it with `python tools/mlarena.py record BL-27 "<x>x tok/s, p95 <ms> ms"`.
  2. Every prompt gets the same greedy output batched as it does alone.
  3. The README has the throughput against p95 plot and names the max_batch and max_wait you would ship.

Side quests

Implement continuous batching, where finished sequences leave the batch and new ones join between decoding steps.
Stream tokens back to the client as they are generated, and measure time to first token separately.

Terms in plain words

Latency
The time from sending one request to getting its full answer back, as the client sees it.
p50 / p95
The median latency, and the latency that 95% of requests beat. p95 shows what your slow users feel.
Throughput
How much work the server finishes per second, counted here in generated tokens per second.
Dynamic batching
Holding requests for a few milliseconds so the server can run several of them through the model in one forward pass.