Fine-tuning stopped being a single-model problem. A team ships one LoRA adapter for contract summarisation, another for the support-triage tone, then a per-customer adapter for each of forty enterprise tenants, and suddenly the serving plan — one replica per fine-tune — needs forty H100s to handle traffic that would comfortably fit on two. That is the failure mode we get called in to fix most often now: the fine-tuning worked, the economics did not.
The fix is multi-tenant adapter serving. One base model stays resident in GPU memory, hundreds of small LoRA adapters are loaded alongside it, and a single batch can contain requests routed to different adapters at the same time. This tutorial covers the mechanics that make that possible, how to turn it on in vLLM, how to train adapters so they are actually swappable, what the latency tax really is, and when you should merge weights instead.
Why a LoRA adapter is cheap to host
LoRA replaces a full weight update with a low-rank one. Instead of storing a modified W of shape [d_out, d_in], you store two thin matrices and compute
y = x @ W.T + (alpha / r) * (x @ A.T) @ B.T
where A is [r, d_in] and B is [d_out, r], with r typically 8 to 64. For an 8B model with adapters on the attention and MLP projections, r = 16 lands around 40–80 MB in BF16. The base model is 16 GB. That ratio is the whole opportunity: you can hold several hundred adapters in the memory footprint of one extra copy of the base weights, and you can hold thousands in host RAM and page them in.
The operational consequence is worth stating plainly. Adapters are data, not deployments. Adding a tenant should be an upload and a registry row, not a Helm release.
The hard part: batching across different adapters
Naive serving destroys the advantage. If request A needs adapter_finance and request B needs adapter_legal, the obvious implementation runs two separate forward passes, and you are back to memory-bandwidth-bound decoding at batch size one — the same trap described in our speculative decoding post.
The technique that fixes it comes from the S-LoRA and Punica line of work: batched grouped GEMM over heterogeneous adapters. The base x @ W.T is one dense matmul for the whole batch as usual. The adapter term is computed by a custom kernel (in vLLM this is a Triton kernel, bgmv/sgmv-style) that takes:
- the batch of hidden states,
- a stacked tensor of all resident
AandBmatrices, - an index vector mapping each sequence in the batch to its adapter slot.
Each token's low-rank correction is gathered from the right slot and added. One kernel launch, one batch, mixed tenants. Adapters with different ranks are handled by padding to the maximum configured rank, which is why --max-lora-rank matters more than you would guess.
Two details bite in production:
- Rank homogeneity pays. A single
r = 64adapter in a fleet ofr = 8adapters forces padding for everyone and inflates the kernel cost. Standardise the rank in your training template. - Target-module homogeneity pays more. If one adapter touches
q_proj, v_projand another touches all seven projections, the serving path has to handle the union. Pick one target-module list per base model and enforce it at training time.
Turning it on in vLLM
The serving side is a handful of flags. Concretely, with an 8B base and adapters trained with PEFT:
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
--enable-lora \
--max-loras 8 \
--max-lora-rank 16 \
--max-cpu-loras 64 \
--lora-modules \
finance=/srv/adapters/finance-v3 \
legal=/srv/adapters/legal-v2 \
--max-model-len 8192 \
--gpu-memory-utilization 0.90
The two numbers people confuse:
--max-lorasis how many adapters can be active in one batch on the GPU. It costs GPU memory and a little kernel overhead. 4–8 is a sane starting point.--max-cpu-lorasis the host-RAM cache. Adapters beyond--max-loraslive here and are swapped in on demand. Make it comfortably larger than your working set.
Requests then select an adapter through the ordinary OpenAI-compatible model field:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
resp = client.chat.completions.create(
model="legal", # the adapter name, not the base model
messages=[{"role": "user", "content": "Summarise the indemnity clause."}],
temperature=0.2,
)
For a fleet that changes daily, static --lora-modules is the wrong shape. Enable the runtime API instead (VLLM_ALLOW_RUNTIME_LORA_UPDATING=1) and register adapters as they land:
import requests
requests.post(
"http://localhost:8000/v1/load_lora_adapter",
json={"lora_name": "tenant-3941", "lora_path": "/srv/adapters/tenant-3941"},
timeout=120,
).raise_for_status()
Wrap that in your control plane so an adapter upload triggers registration on every replica, and keep a /v1/unload_lora_adapter call in the same code path for deprecation. Treat the adapter directory as immutable and versioned — tenant-3941/v4, never tenant-3941/latest — or you will eventually serve two different behaviours from two replicas and spend a day not reproducing it.
Training adapters that are safe to co-serve
Most multi-tenant serving incidents are really training-time mistakes. The rules we hand clients:
from peft import LoraConfig
# One template for every tenant on a given base model.
config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
)
- Pin the base model by revision. An adapter is a delta against exact weights.
Meta-Llama-3.1-8B-Instructat one commit is not the same tensor as another. Record the base repo and revision in the adapter's metadata and refuse to load a mismatch. - Do not train the embeddings or the LM head.
modules_to_savewithembed_tokensorlm_headproduces an adapter that cannot share a batch cheaply, and in vLLM it needs extra flags at best. If a tenant genuinely needs new tokens, that tenant needs its own deployment. - Never mix tokenizers. All tenants on a replica share one tokenizer. Adding special tokens per tenant breaks that assumption silently.
- Keep QLoRA for training only. Training the adapter against a 4-bit base is fine and cheap — see our QLoRA walkthrough — but serve it against a base whose quantisation you have actually evaluated, and re-run your evals after changing base precision. Our precision guide covers that trade.
What it costs you
Numbers vary by hardware and model, but the shape is consistent across engagements on 8B-class models:
| Setup | Throughput vs base | Notes |
|---|---|---|
| Base model, no LoRA | 1.00x | Reference |
--enable-lora, 1 active adapter | 0.90–0.97x | Kernel overhead only |
| 4–8 active adapters in one batch | 0.85–0.95x | Padding + gather cost |
| Adapter swap from CPU cache | +20–100 ms one-off | PCIe copy on first request |
| One replica per adapter | 1.00x each | And Nx the GPUs |
So you pay roughly 5–15% throughput to collapse N deployments into one. At N = 3 that is already a large win; at N = 40 it is the difference between a viable product and a dead one.
The pathology to watch for is swap thrashing: if your active set genuinely exceeds --max-loras on every batch, adapters evict each other and per-request latency gets spiky. The fix is routing, not bigger flags — shard tenants across replicas by adapter affinity (a consistent hash on adapter name in front of the fleet) so each replica serves a stable working set. This composes cleanly with the queue-depth-based scaling described in our GPU autoscaling guide.
When to merge instead
Multi-tenant serving is not always right. Merge the adapter into the base weights (merged = model.merge_and_unload()) when:
- One adapter serves effectively all of your traffic. Then you are paying the LoRA kernel tax for nothing.
- You need the last few percent of latency, or you are exporting with
torch.export/AOTInductor for a Python-free serving path — see AOTInductor in production. A merged model is a plain model and compiles like one. - You are quantising aggressively after fine-tuning; quantise the merged weights and evaluate once, rather than trying to reason about a low-rank correction on top of INT8.
And keep adapters unmerged when tenants are many, churn is high, or you need per-tenant rollback. Those are governance properties as much as performance ones.
Observability you will want on day one
Three things, all cheap to add, all painful to retrofit:
- Per-adapter request counts, token counts and p95 latency. Multi-tenant serving means one noisy tenant is now everyone's problem; you need to be able to name it.
- Adapter swap rate and cache hit rate. This is your early warning for thrashing, and it is the metric that tells you when to add a replica or change the routing key.
- Per-adapter eval scores, re-run on every base-model change. A base upgrade is a fleet-wide behaviour change. Score every adapter against its own golden set before promoting, using the kind of harness described in building an LLM evaluation harness.
A rollout plan
- Standardise the LoRA training template — rank, alpha, target modules, pinned base revision — and re-train any adapter that does not conform.
- Stand up one vLLM replica with
--enable-lora, two adapters, and a load test that mixes them in the same batch. Confirm the throughput tax matches the table above. - Add the runtime load/unload API behind your control plane, with immutable versioned adapter paths.
- Add per-adapter metrics and per-adapter evals before onboarding tenant number three.
- Introduce adapter-affinity routing once your active set approaches
--max-loras. - Revisit merging annually, or whenever one adapter's share of traffic passes ~80%.
Most teams can complete steps 1–3 in a week and cut their inference fleet by more than half in doing so.
We do this work with clients regularly: consolidating per-customer fine-tunes onto shared replicas, designing the adapter registry and routing, and measuring the quality of each tenant before and after. See our LLM fine-tuning and customization and LLM inference and serving optimization services, or contact us with your adapter count and latency SLO and we will tell you how many GPUs you should actually need.