Most teams size their inference cluster on parameters and throughput, then get blindsided by the KV cache. A 70B model in BF16 needs ~140 GB for weights, which is a one-time cost you can plan around. The KV cache is the part that scales with traffic: every concurrent request holds a tensor proportional to its context length, and long-context workloads (100k-token codebases, 60-page contracts, month-long agent transcripts) make that cache, not the weights, the thing that decides how many users fit on a GPU.
This tutorial is about the three levers that actually move long-context serving cost: prefix caching, KV cache offload and tiering, and prefill/decode disaggregation. It also covers context parallelism for the training-side version of the same squeeze. Everything here is measurable from your own logs before you buy a single extra GPU.
First, size your KV cache
The arithmetic is simple and worth doing by hand once, because it is usually where the surprise lives:
kv_bytes = 2 (K and V)
* num_layers
* num_kv_heads * head_dim
* dtype_bytes
* seq_len
For a Llama-3.3-70B-shaped model with 80 layers, 8 KV heads (GQA) and head_dim 128 in BF16:
per_token = 2 * 80 * 8 * 128 * 2 = 327,680 bytes (~0.31 MiB/token)
So one 128k-token request holds about 40 GiB of KV. On an 8x80GB node with ~140 GB of weights and framework overhead, you have roughly 480 GB of KV headroom: about twelve concurrent 128k requests, or a few hundred 4k chat turns. That single number explains most "why is our long-context endpoint so expensive" tickets.
A quick script to run against your own config:
def kv_gib(layers, kv_heads, head_dim, seq_len, dtype_bytes=2, batch=1):
per_tok = 2 * layers * kv_heads * head_dim * dtype_bytes
return per_tok * seq_len * batch / (1024 ** 3)
print(kv_gib(80, 8, 128, 128_000)) # ~39.1 GiB
print(kv_gib(80, 8, 128, 8_000, batch=64)) # ~156 GiB
Two levers shrink it before you touch the serving topology at all: GQA/MQA (already baked into modern checkpoints) and KV quantization to FP8, which halves the cache for a usually-negligible quality cost. Measure the second one on your eval set rather than trusting a blog post — including this one.
Lever 1: prefix caching (the cheapest win in LLM serving)
Prefill is compute-bound and scales roughly linearly in prompt tokens. If every request in your workload starts with the same 6k-token system prompt, tool schema, or retrieved document set, you are re-computing identical attention keys and values thousands of times a day.
Prefix caching (also called automatic prefix caching, or APC) hashes prompt blocks and reuses the KV blocks of any prefix that has been seen before. In vLLM:
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 4 \
--max-model-len 131072 \
--enable-prefix-caching \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.92
The engineering work is not the flag; it is making your prompts cache-friendly. Rules that repeatedly pay off:
- Put everything stable at the front: system prompt, tool definitions, few-shot examples, policy text. Put the user turn and anything per-request last.
- Never inject a timestamp, request ID, or randomized instruction ordering near the top of the prompt. One volatile token at position 12 invalidates the entire prefix behind it.
- Order retrieved RAG chunks deterministically (by document ID, not by a float score that jitters), so repeat queries hit the cache.
- For agent loops, append to a growing transcript rather than rebuilding it; each turn then extends a cached prefix instead of creating a new one.
Typical results on document-QA and agent traffic: 40-80% reduction in prefill FLOPs and a step-change in time-to-first-token on warm prefixes. Watch the prefix_cache_hit_rate gauge; if it is under 30% on repetitive traffic, your prompt template is the bug.
Lever 2: KV offload and cache tiering
Prefix caching only helps while the blocks are still resident in GPU memory. Under pressure the engine evicts them, and the next request re-prefills from scratch. Tiering fixes that by pushing cold KV blocks down a hierarchy — HBM, then host DRAM over PCIe, then NVMe or a shared cache service — and pulling them back on a hit.
The trade is bandwidth versus compute. Re-loading a block from host DRAM costs a PCIe transfer; re-computing it costs a full prefill pass over those tokens. For long prefixes the transfer wins by a wide margin, which is why KV offload layers (LMCache and similar, plus the CPU-offload options built into the major engines) have become standard for multi-turn and multi-tenant deployments.
A back-of-envelope test before you adopt one:
recompute_ms ~= prefix_tokens / prefill_tokens_per_sec * 1000
reload_ms ~= prefix_tokens * per_token_kv_bytes / effective_pcie_bytes_per_sec * 1000
If reload_ms is comfortably below recompute_ms at your prefix lengths, tiering is worth the operational complexity. It usually is above ~8k tokens of reusable prefix and rarely is below ~2k.
Operational notes: cap the host-DRAM tier explicitly so the KV cache cannot compete with your dataloader for page cache; give the offload store its own eviction metrics; and make cache keys include the model revision, quantization and tokenizer version, or a redeploy will serve KV blocks computed by a different model.
Lever 3: prefill/decode disaggregation
Prefill and decode are different machines wearing the same uniform.
| Prefill | Decode | |
|---|---|---|
| Bottleneck | Compute (FLOPs) | Memory bandwidth |
| Parallelism | Large, one pass over many tokens | One token per step per sequence |
| Ideal hardware | Highest FLOPs available | Highest HBM bandwidth and capacity |
| Latency metric | TTFT | TPOT |
Run them in one pool and they interfere: a 100k-token prefill occupies the GPU long enough to stall every decode step behind it, so your p99 inter-token latency spikes whenever a long document arrives. Chunked prefill softens this by slicing prefill into pieces interleaved with decode, and it should be your first move because it is a config change.
Disaggregation is the structural fix: a prefill pool computes the KV cache and hands it to a decode pool over a fast interconnect, each pool scaled and tuned independently. This is how the large-scale 2026 stacks (vLLM's disaggregated serving paths, NVIDIA Dynamo, Mooncake-style architectures) hit tight TTFT and tight TPOT simultaneously.
What it buys you:
- Independent scaling: long-document traffic adds prefill nodes only.
- Clean SLOs: a burst of 100k-token prompts no longer jitters ongoing conversations.
- Hardware fit: cheaper high-FLOP parts for prefill, high-bandwidth/high-capacity parts for decode.
What it costs:
- A KV transfer path that must not become the bottleneck — plan on RDMA/NVLink-class links, not plain TCP.
- A second scheduler, more failure modes, and far more moving parts to observe.
- Real benefit only above a few nodes. Below that, chunked prefill plus prefix caching gets most of the win.
Rule of thumb: single node, mixed traffic → chunked prefill + prefix caching. Multi-node with bimodal prompt lengths and strict TTFT/TPOT SLOs → disaggregate.
The training-side twin: context parallelism
The same KV pressure appears in training when sequence length grows: activations and attention state per sequence outgrow one device, and no amount of FSDP sharding of parameters helps, because the problem is the sequence. Context parallelism splits the sequence dimension across ranks and exchanges K/V shards ring-fashion during attention.
In PyTorch this composes with FSDP2 and tensor parallelism through DeviceMesh:
import torch
from torch.distributed.device_mesh import init_device_mesh
# 8 GPUs: 2-way context parallel x 4-way data parallel
mesh = init_device_mesh("cuda", (4, 2), mesh_dim_names=("dp", "cp"))
cp_mesh = mesh["cp"]
# fully_shard(model, mesh=mesh["dp"]) as usual, then run the attention
# region under the context-parallel context manager so K/V shards are
# exchanged across cp_mesh instead of being materialized per rank.
The context-parallel APIs are still moving between PyTorch releases (they live under torch.distributed.tensor.experimental / the context_parallel helpers), so pin your version and read the docs for that exact tag. The mental model is stable: CP for sequence length, TP for layer width, FSDP for parameter memory, PP for depth. Reach for CP only after activation checkpointing and a memory-efficient attention kernel have failed to fit the sequence you need.
Metrics that tell you which lever to pull
Instrument these before changing topology:
- Prompt length distribution (p50/p95/p99). Bimodal distributions are the signal for disaggregation.
- Prefix cache hit rate, plus mean cached tokens per hit.
- TTFT vs prompt length — a steep slope means prefill-bound.
- TPOT under concurrency, not at batch size one.
- KV cache utilization and preemption/recompute counts. Frequent preemption means you are over-admitting; cap
max_num_seqsrather than letting the scheduler thrash. - Tokens/sec/GPU/$ for the whole pool. It is the only number finance cares about, and the only one that catches a "win" that quietly halved throughput.
A practical rollout order
- Compute your KV bytes per token and per-request ceiling. Decide the max context you will actually support and enforce it.
- Restructure prompts so the stable part is a genuine shared prefix. Turn on prefix caching. Measure the hit rate.
- Enable FP8 KV cache; validate quality on your eval harness before keeping it.
- Turn on chunked prefill and re-check p99 TPOT.
- Add a DRAM/NVMe KV tier if your reusable prefixes exceed a few thousand tokens and eviction is hurting hit rate.
- Only then consider prefill/decode disaggregation, and only if you are already multi-node with distinct TTFT and TPOT SLOs.
Long-context serving costs are not a fixed property of the model; they are a property of how much redundant KV you compute and how well you place the KV you keep. The first three steps above are configuration and prompt hygiene, and they routinely cut long-context serving cost by half. The heavier machinery is worth it later, on evidence — never as step one.