Autoscaling GPU Inference on Kubernetes: Cold Starts, Scale-to-Zero and Cost per Million Tokens

Most teams get their PyTorch model fast and then discover the expensive part: keeping it available. A single H100 replica idling at 8% utilisation costs the same per hour as one saturated at 95%, and the usual reflex — pin four replicas and forget about it — is how a $400/month inference workload turns into a $12,000/month line item. The fix is autoscaling. The reason autoscaling gets abandoned is that GPU pods take three to eight minutes to become useful, so scaling up arrives after the traffic spike and scaling down causes user-visible errors.

This tutorial is the playbook we use with clients: which signal to scale on (it is not GPU utilisation), how to get a cold start from minutes to seconds, when scale-to-zero actually pays, and how to turn all of it into a defensible cost per million tokens.

Why CPU and GPU utilisation are the wrong signals

The default Kubernetes HPA scales on CPU. For an inference server the CPU is mostly idle, so nothing ever scales. The obvious upgrade is DCGM_FI_DEV_GPU_UTIL from the DCGM exporter, and it is almost as bad: that metric reports the fraction of time any kernel was resident, not how much work got done. A server decoding one token at a time at batch size one will happily report 90%+ while the SMs are starving on memory bandwidth. You will scale out a replica that is not busy and never scale out the one that is.

Scale on the thing your users feel, which for a continuous-batching LLM server is queue pressure:

Workload	Signal	Typical target
LLM server (vLLM, SGLang, TGI)	vllm:num_requests_waiting	scale up above ~2–5 waiting
LLM server, latency SLO	running requests / max concurrency	0.6–0.8
Encoder, ranker, embeddings	in-flight requests per replica	batch size that hits p99 SLO
Batch/offline scoring	queue depth in SQS/Kafka/Redis	fixed messages-per-replica

Every serious serving runtime exposes these on /metrics. With Prometheus and KEDA the scaler is a few lines:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-gateway
spec:
  scaleTargetRef:
    name: llm-gateway
  minReplicaCount: 1          # 0 only if you have read the cold-start section
  maxReplicaCount: 12
  cooldownPeriod: 300         # do not thrash expensive pods
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: |
          sum(vllm:num_requests_waiting{service="llm-gateway"})
          / sum(up{service="llm-gateway"})
        threshold: "3"

Two rules that save a lot of money and a lot of pages:

1. Scale up fast, scale down slow. Use an HPA behavior block (or KEDA's cooldownPeriod plus stabilizationWindowSeconds) with something like 0s stabilisation up and 300–600s down. A GPU pod that flaps costs more in cold starts than the idle minutes you saved.
2. Never scale on an average across replicas without also capping per-replica concurrency. Otherwise one wedged pod absorbs a queue and hides the signal.

The cold-start budget

Measure it before you tune it. Instrument the pod from scheduled to first successful readiness probe and break it into five buckets. A realistic untuned 70B-class deployment looks like this:

Phase	Untuned	Tuned
Node provisioning (cluster autoscaler)	120–240s	0s (warm pool / spare capacity)
Image pull (CUDA + torch + runtime ≈ 12 GB)	90–180s	10–25s
Weight download from object storage	60–300s	15–40s
Runtime init: CUDA context, graph capture, warm-up	60–120s	10–30s
Readiness + load-balancer registration	5–30s	5s

Each phase has a known fix.

Node provisioning. This is usually the single biggest term and the one nobody measures. Either keep headroom with low-priority placeholder pods that real workloads preempt, or use a provisioner (Karpenter, or your cloud's node auto-provisioning) with a warm pool. Headroom is cheap relative to a queue of timed-out requests; one idle A10G is a rounding error next to a 4-minute outage during a demo.

Image pull. Split the fat base image so the CUDA and PyTorch layers are shared and cached on the node, keep application layers tiny, and enable lazy pulling — eStargz/SOCI let the container start before the whole image is resident. Do not bake multi-gigabyte weights into the image; layers that change with every model version defeat every cache you have.

Weight download. Stream from object storage with parallel range requests, and keep a node-local cache (a hostPath or local NVMe PVC) keyed by model revision. With safetensors and mmap the load is essentially a page-in, so the bottleneck is network throughput: on a 25 Gbps node, 140 GB of FP8 weights is ~60s at best, which is why most teams keep at least one warm replica per model instead.

Runtime init. This is where PyTorch-side work pays off. CUDA graph capture and torch.compile can add a minute per pod if they re-run on every start. Persist the Inductor cache to a shared volume (TORCHINDUCTOR_CACHE_DIR), or remove the problem entirely by compiling ahead of time — see serving PyTorch without Python with AOTInductor for non-LLM models, and set VLLM_DISABLE_COMPILE_CACHE=0 plus a persistent cache dir for LLM servers. Run exactly one warm-up request shaped like real traffic before signalling readiness.

readinessProbe:
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 5
  failureThreshold: 3
startupProbe:                 # give the model time without loosening readiness
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 10
  failureThreshold: 60        # up to 10 minutes to load
terminationGracePeriodSeconds: 120
lifecycle:
  preStop:
    exec: { command: ["sleep", "20"] }   # drain before SIGTERM

The preStop sleep and a long grace period are not optional. Scale-down without draining is the most common cause of "autoscaling broke our API": the pod is removed from the endpoints list and killed at the same moment, and every in-flight generation — which may be 30 seconds long — dies with it.

Scale-to-zero: when it pays and when it hurts

Scale-to-zero is right for internal tools, demo environments, per-tenant fine-tuned models with sporadic traffic, and batch jobs. It is wrong for anything with a user-facing latency SLO and unpredictable arrivals, unless you can hide the cold start behind a queue and a "working…" state.

The arithmetic is simple. Let C be the GPU cost per hour, h the idle hours you would eliminate per day, and n the number of cold starts per day at t seconds each. Scale-to-zero saves C * h per day and costs you n degraded requests. On an $11/hr H100 node with 14 idle hours a day that is ~$150/day saved — worth a lot of engineering. On a $1.10/hr A10G with 6 idle hours it is $6.60/day, and not worth one paged engineer.

Patterns that make zero survivable:

• Queue first, GPU second. Accept the request into a queue, return a job id, scale the consumer on queue depth. Users see a progress state instead of a 60-second hang. This is also the cheapest way to run document extraction with a VLM or any bulk pipeline.
• Multi-tenant LoRA instead of multi-deployment. If you serve fifty fine-tunes of the same base model, do not run fifty deployments. Serve one base with hot-swappable adapters (vLLM's --enable-lora) so the expensive weights stay resident and per-tenant scaling becomes free. See fine-tuning Llama with QLoRA for how those adapters are produced.
• Tiered fallback. Keep one small always-on model for the first response and scale the big one behind it.
• Scheduled prescaling. Traffic is rarely random. A CronJob that sets minReplicaCount to 3 at 08:00 local and back to 0 at 20:00 beats any reactive scaler for business-hours products.

Getting more out of each GPU before adding one

Autoscaling should be the last lever, not the first. In most reviews we find 2–4x of headroom inside the replicas that are already running:

• Continuous batching — non-negotiable for LLMs. See serving an LLM with vLLM.
• Right-sized precision. FP8 weights and KV cache roughly double the concurrency a card can hold; the trade-offs are in BF16, FP8, INT8.
• KV cache discipline. Prefix caching and sane max_model_len decide how many sessions fit. Long-context serving economics has the numbers.
• Fractional GPUs for small models. MIG partitions on A100/H100, or time-slicing via the NVIDIA device plugin, let five ranker models share one card instead of holding five.
• Separate prefill and decode pools. They have opposite bottlenecks; scaling them together means one is always wrong.
• A smaller model. Distillation or speculative decoding often removes the capacity problem outright.

Turn it into cost per million tokens

Autoscaling without a unit-economics metric is guesswork. Export tokens from your serving runtime and node-hours from your cloud bill, then track one number per model per day:

# $ per 1M output tokens, per model, over 24h
(
  sum by (model) (
    avg_over_time(kube_pod_container_resource_requests{resource="nvidia_com_gpu"}[24h])
  ) * 24 * on() group_left() gpu_hourly_cost
)
/
(
  sum by (model) (increase(vllm:generation_tokens_total[24h])) / 1e6
)

Plot it next to p95 latency. Every change — a new scaler threshold, FP8, a bigger max_num_seqs, scale-to-zero — should move one of those two lines in the right direction without wrecking the other. A team that can say "we serve 40M tokens a day at $1.80 per million at p95 of 900ms" can make capacity decisions in a meeting; a team that can only say "the GPU bill went up" cannot. Our GPU budgeting guide covers the buy-versus-rent side of the same question.

A rollout that does not page anyone

1. Instrument first: queue depth, in-flight requests, token counters, cold-start histogram. One week of baseline data.
2. Add the HPA/KEDA scaler with minReplicas at today's fixed count and maxReplicas higher. Only scaling up is enabled in practice; nothing can get worse.
3. Fix draining — preStop, grace period, and a load test that kills pods mid-generation and asserts zero 5xx.
4. Attack cold start until p90 is under 60s. Measure each phase; do not guess.
5. Lower minReplicas one step at a time, watching p99 and error rate for a few days at each step.
6. Consider zero only for workloads whose users tolerate the first-request penalty, and only after step 4.

Steps 3 and 4 are the ones teams skip, and they are the reason the autoscaler gets disabled two weeks later after an incident.

We help teams do exactly this work — scaler design, cold-start surgery, capacity models and the serving-side optimisation that makes the whole thing cheaper. See our PyTorch cloud deployment and PyTorch performance optimization services, or contact us with your traffic shape and latency SLO and we will tell you what your fleet should actually cost.