Serving an LLM with vLLM: Zero to Production

vLLM is the default way to serve open-weight large language models in 2026: it takes a Hugging Face model directory, exposes an OpenAI-compatible HTTP API, and uses paged attention and continuous batching to get an order of magnitude more throughput from a GPU than a naive model.generate loop. This tutorial goes from a fresh machine to a production-shaped deployment: install, serve a quantized model, understand batching with a load test, tune the two settings that matter, add observability, and scale.

1. Install and serve

You need a CUDA GPU with a recent driver and Python 3.10+. vLLM ships pre-built wheels pinned to a specific PyTorch version, so install it in a clean environment and let it bring its own torch:

python -m venv .venv && source .venv/bin/activate
pip install vllm

Serve a quantized 8B model. AWQ and GPTQ 4-bit checkpoints for the popular open-weight families are published on the Hugging Face Hub; FP8 checkpoints are the better choice on Hopper-class and newer GPUs.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --port 8000

--quantization fp8 quantizes a BF16 checkpoint on load; for a pre-quantized AWQ checkpoint pass that repository name instead and vLLM detects the format. The first start downloads weights and compiles CUDA graphs; subsequent starts are faster. Check it is up:

curl localhost:8000/v1/models

2. The OpenAI-compatible API

Any OpenAI client works by pointing base_url at the server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Summarize paged attention in two sentences."}],
    max_tokens=120,
    temperature=0.2,
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Streaming is the default you want for chat interfaces: time-to-first-token (TTFT) is what the user perceives, and it is far lower than total generation time. Tool calling, JSON-schema constrained output (response_format) and logprobs are all supported through the same endpoint; check the vLLM docs for the current flags per model family.

3. Continuous batching, demonstrated

The reason vLLM is fast is that it does not wait for a batch to finish before admitting new requests. Each decode step, the scheduler adds any newly arrived sequences and drops finished ones, so the GPU always works on as many sequences as the KV cache can hold. Paged attention is what makes that possible: KV cache is allocated in fixed-size blocks like virtual memory, so sequences of different lengths share the GPU without fragmentation.

See it with a load test. Save this as load.py:

import asyncio, time
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="unused")
PROMPT = "Write a 150-word product description for a mechanical keyboard."


async def one():
    t0 = time.perf_counter()
    first = None
    tokens = 0
    stream = await client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": PROMPT}],
        max_tokens=200, stream=True)
    async for chunk in stream:
        if chunk.choices[0].delta.content:
            tokens += 1
            first = first or time.perf_counter() - t0
    return first, tokens, time.perf_counter() - t0


async def main(concurrency):
    t0 = time.perf_counter()
    results = await asyncio.gather(*(one() for _ in range(concurrency)))
    wall = time.perf_counter() - t0
    ttft = sorted(r[0] for r in results)
    total_tokens = sum(r[1] for r in results)
    print(f"concurrency {concurrency:4d}  p50 TTFT {ttft[len(ttft)//2]*1000:6.0f} ms  "
          f"p95 TTFT {ttft[int(len(ttft)*0.95)]*1000:6.0f} ms  "
          f"throughput {total_tokens/wall:7.0f} tok/s")


for c in (1, 8, 32, 128):
    asyncio.run(main(c))

Run it and watch aggregate throughput climb almost linearly with concurrency while per-request TTFT grows slowly, until the KV cache fills and requests start queuing. That knee is your capacity for this model on this GPU at this context length.

4. The two settings that matter

--max-num-seqs caps how many sequences the scheduler runs concurrently. Raise it until throughput stops improving or p95 TTFT exceeds your SLO; lower it for a latency-first deployment.

--max-model-len and --gpu-memory-utilization set the KV-cache budget. Memory not used by weights is KV cache; a longer maximum context means fewer concurrent sequences fit. Set max-model-len to the longest request you actually serve, not the model's theoretical maximum. The startup log prints the number of KV blocks and the maximum concurrency at full context; read it.

Two more worth knowing: --enable-prefix-caching (on by default in current releases) reuses KV cache for shared prompt prefixes, which is a large win for system-prompt-heavy chat workloads; and --speculative-config with a small draft model can cut decode latency for latency-critical paths at some throughput cost.

5. Observability

vLLM exposes Prometheus metrics at /metrics. The ones to put on a dashboard:

• vllm:num_requests_running and vllm:num_requests_waiting: queueing means you are at capacity.
• vllm:gpu_cache_usage_perc: KV-cache utilization; sustained near 100% means raise memory or lower context.
• vllm:time_to_first_token_seconds and vllm:time_per_output_token_seconds histograms: your user-facing SLOs.
• vllm:prompt_tokens_total and vllm:generation_tokens_total: the basis of cost per token.

Log request IDs through to your application so a slow user request can be correlated with server-side queue depth at that moment.

6. Production shape

Run vLLM in its official Docker image behind a load balancer with health checks on /health. For models that do not fit on one GPU, --tensor-parallel-size N splits them across N GPUs on one host. For more throughput than one host provides, run replicas and scale on num_requests_waiting rather than GPU utilization, which is always near 100% under load. On Kubernetes, give each replica a warm-up request in its readiness probe so a new pod does not take traffic before CUDA graphs are captured.

Put an API gateway in front for authentication, rate limiting and per-tenant token accounting; vLLM itself does only an optional static API key. And pin the vLLM version in your image: releases are frequent and occasionally change default flags.

If you are fine-tuning the model you serve, see Fine-tuning Llama with QLoRA for producing merged weights vLLM loads directly. For help sizing and tuning a deployment, including cost-per-token modelling, see our LLM inference and serving optimization service or contact us.