Why is my LLM server slow? Three experiments on memory, batching and context
Why is my LLM server slow? Three experiments on memory, batching and context
You ask an LLM server for a story and tokens appear at some speed. What decides that speed? How many users can share one GPU? What does a “long context” cost?
I tried to answer these with measurements instead of folklore. I ran a small model (Gemma 3 4B, 4-bit quantized, 2.4 GB) on one machine (an NVIDIA GB10 with 273 GB/s of unified memory) using llama.cpp’s llama-server. For each experiment I wrote a prediction first and then checked it. The full script is lab1_experiments.py, and it needs only Python’s standard library.
Background: two phases, two bottlenecks
Generating text has two phases:
- Prefill: the model reads your prompt, all tokens at once. This is compute-heavy.
- Decode: the model produces one token at a time. For every token it must read all the weights from memory, plus the KV cache (stored attention state for every earlier token).
Decode is where most of the waiting happens, and it is usually limited by memory bandwidth, not by arithmetic. Keep that in mind: it explains all three results.
A quick check. A single request ran at about 70 tokens/s. Each token reads ~2.3 GiB of weights, so 70 x 2.3 GiB is about 165 GB/s, which is more than half of the machine’s 273 GB/s peak. The GPU is mostly waiting for memory.
A measurement trap before we start
My first test asked for max_tokens: 4000 and finished in 10 seconds with 744 tokens. max_tokens is only a ceiling. The model emitted its end-of-text token and stopped.
For benchmarks, that breaks the experiment, because the output length is no longer under your control. Fix: send ignore_eos: true so the model cannot stop early, and always check completion_tokens in the response. Every number below uses this, plus temperature: 0 and cache_prompt: false.
Experiment 1: what does a bigger context window cost in memory?
Prediction. Weights are fixed. The KV cache grows with the context you reserve (-c). For each token and each layer, the cache stores a key and a value: 2 x 4 kv_heads x 256 head_dim x 2 bytes = 4096 bytes. Gemma 3 is unusual: only 5 of its 34 layers look at the whole context. The other 29 use a sliding window of 1024 tokens, so their cache does not grow. That means about 5 x 4096 B = 20 KiB per extra token of context.
Result (GPU memory of the server process):
-c | memory | growth | predicted |
|---|---|---|---|
| 8,192 | 2983 MiB | ||
| 32,768 | 3487 MiB | +504 MiB | 480 MiB |
| 65,536 | 4159 MiB | +672 MiB | 640 MiB |
The prediction was within 5%.
What to take away.
- You pay for the context you reserve, not the context you use. Setting
-c8x too high wastes memory for nothing. - Architecture matters a lot. Without the sliding window, the same 65k context would need about 4.25 GiB of KV cache instead of ~0.7 GiB.
- A pitfall: the process’s RSS (what
topshows) barely moved (+26 MiB), because GPU allocations are not counted in it on this machine. Check which memory number you are reading.
Experiment 2: what happens with many users at once?
Prediction. If decode is limited by reading weights, then serving 8 requests in one batch should read the weights once per step, not 8 times. Total throughput should grow almost linearly at first while each user slows down only a little. Beyond the server’s number of slots, extra requests must queue.
Result (8 slots, 200 tokens per request):
| concurrent requests | total tok/s | tok/s per request | worst time to first token |
|---|---|---|---|
| 1 | 54 | 54 | 0.03 s |
| 2 | 137 | 69 | 0.06 s |
| 4 | 174 | 44 | 0.07 s |
| 8 | 382 | 49 | 0.15 s |
| 12 | 324 | 53 | 4.19 s |
What to take away.
- Going from 1 to 8 requests multiplied total throughput by about 7, while each user lost only 10-30% of their speed. Eight users cost far less than eight times one user. This is why every serious LLM server batches requests.
- At 12 requests, only 8 fit in slots. The other 4 wait for a free slot, so their first token arrives after ~4.2 s. No GPU slowdown, just a queue. Latency under load often shows up as waiting, not as slow decoding.
- Total throughput fell at 12 because the second wave had only 4 requests, so the batch was no longer full.
- Honesty about noise: with 200 tokens per request, each run lasts only 3-4 s, and rerunning the same script moved some rows by 10-40% (N=2 beating N=1 is noise, not physics). Read the shape of the table, not single cells. A proper benchmark would use longer outputs and report the median of several repeats.
Experiment 3: what happens at the end of the context?
Prediction. The context size -c is shared among slots: each request gets -c / parallel tokens, and prompt plus output must fit inside. With -c 8192 and one slot, a 10-token prompt should stop after about 8182 output tokens even if I ask for 9000. Decode should also get slightly slower as the KV cache grows, but only slightly: at 8k tokens the cache is ~0.27 GiB, compared with 2.3 GiB of weights, so at most ~10% slower.
Result.
- Prompt 10 + output 8182 = exactly 8192 tokens. The server stopped with
finish_reason: "length". - Decode speed over the generation:
| output tokens | tok/s |
|---|---|
| 1-500 | 70.9 |
| 500-1000 | 69.9 |
| 1000-2000 | 68.4 |
| 2000-4000 | 68.0 |
| 4000-6000 | 66.7 |
| 6000-8000 | 66.3 |
What to take away.
finish_reason: "length"is ambiguous. It means either “hitmax_tokens” or “slot full”. To tell them apart, compareprompt_tokens + completion_tokenswith the slot size (/propsshows it).- Never trust the flag alone: llama.cpp can raise a too-small
-c, so confirm the real per-slot size. - Speed fell only 6.5% across 8,000 tokens. At this scale a long context costs memory (Experiment 1) much more than speed. The speed cost grows with the context, so it matters at 100k+ tokens or when many long requests run together.
Summary: a mental model you can reuse
- Memory = weights + KV cache. The KV cache scales with the context you reserve, and the model’s architecture changes the constant a lot.
- Single-request decode is memory-bandwidth bound. Tokens/s is roughly bandwidth divided by bytes read per token.
- Batching is nearly free until compute takes over. That is where throughput per dollar comes from, and where queueing starts when slots run out.
- Context is split across slots. More parallel users means less context each, unless you add memory.
- Measure with controlled output length (
ignore_eos, checkcompletion_tokens) and look at the right memory metric.
Try it yourself
python3 lab1_experiments.py # all three
python3 lab1_experiments.py memory context # just some
It starts its own temporary server, so edit BIN, MODEL and PORT at the top for your machine. Then change one thing and predict before you run: a different --parallel, a different model, an 8-bit KV cache. Your numbers will differ, and why they differ is the lesson.
Setup: GB10 (aarch64, unified memory), llama.cpp llama-server, gemma-3-4b-it-Q4_K_M.gguf, flags -ngl 99 -fa on. Single machine, small sample sizes, so treat results as illustrations of the mechanisms, not benchmarks.