Most PyTorch serving advice is about the interactive path: keep p99 latency down, keep the GPU warm, autoscale when traffic spikes. But a large share of the real GPU spend in production ML has no user waiting on it at all. Re-embedding a 40-million-document corpus after an embedding model upgrade. Running an extraction LLM over ten years of scanned contracts. Scoring every account in the warehouse nightly. Generating synthetic training data. Backfilling classifications after a schema change.
These are offline batch inference jobs, and they obey completely different economics than online serving. There is no latency SLO, only a deadline and a budget. That one difference lets you make choices an online endpoint can never make: enormous batch sizes, aggressive quantization, cheap preemptible hardware, and 100% GPU saturation. Teams that run batch jobs on their online serving stack routinely pay three to ten times more than they need to.
This tutorial covers how to build batch inference that finishes on time and does not fall over: throughput-first configuration, sorting by length, checkpointing for preemption, spot instances, and the orchestration patterns that make a 20-hour job restartable.
Rule one: stop thinking in requests
An online endpoint processes a stream of arrivals it cannot control. A batch job owns the whole workload up front. That means you can reorder it, and reordering is the single biggest free win available.
With variable-length inputs, a batch's cost is set by its longest member. Random batching of mixed-length text wastes most of the padding budget. Sorting by token length before batching (sometimes called length bucketing) commonly delivers 1.5x to 3x throughput on encoder models with zero quality impact:
import torch
from torch.utils.data import DataLoader
lengths = [len(tok(d["text"])["input_ids"]) for d in docs]
order = sorted(range(len(docs)), key=lambda i: lengths[i])
sorted_docs = [docs[i] for i in order] # keep `order` to restore output alignment
def collate(batch):
return tok([b["text"] for b in batch], padding=True,
truncation=True, max_length=512, return_tensors="pt")
loader = DataLoader(sorted_docs, batch_size=256, collate_fn=collate,
num_workers=8, pin_memory=True, prefetch_factor=4)
Two details matter. First, keep the permutation so you can map results back to source IDs; silently returning rows in a different order than they arrived is a classic batch-pipeline bug. Second, if you sort perfectly you lose all shuffling, which is fine for inference but fatal if you reuse the same code path for training.
For an encoder or classifier, a throughput-shaped inference loop looks like this:
model = model.eval().to("cuda", dtype=torch.bfloat16)
model = torch.compile(model, mode="max-autotune", dynamic=True)
results = []
with torch.inference_mode():
for batch in loader:
batch = {k: v.to("cuda", non_blocking=True) for k, v in batch.items()}
out = model(**batch)
results.append(out.logits.float().cpu())
torch.inference_mode() rather than no_grad() (it also skips version counter bookkeeping), bfloat16 weights, non_blocking=True with pinned memory so H2D copies overlap compute, and dynamic=True on the compile so that varying sequence lengths do not trigger a recompile for every bucket. Pad your bucket lengths to multiples of 64 or 128 to keep the number of compiled shapes small.
Rule two: for LLMs, use an offline engine, not an HTTP endpoint
If your batch job calls your own online inference server over HTTP, you are paying for request serialization, tokenizer round-trips, connection overhead and, worst of all, a scheduler tuned to protect latency rather than to maximize tokens per second.
Use the offline entry point of your inference engine instead. In vLLM that is LLM.generate, which takes the full list of prompts and runs continuous batching over them with no network in the loop:
from vllm import LLM, SamplingParams
llm = LLM(
model="Qwen/Qwen3-8B",
quantization="fp8",
max_model_len=8192,
gpu_memory_utilization=0.92, # higher than you would dare online
max_num_seqs=512, # deep queue: throughput over latency
enable_prefix_caching=True, # big win for shared system prompts
tensor_parallel_size=1,
)
params = SamplingParams(temperature=0.0, max_tokens=512)
outputs = llm.generate(prompts, params)
The settings worth arguing about:
gpu_memory_utilization: online you leave headroom for traffic spikes. Offline there are no spikes. Pushing to 0.90-0.95 buys KV cache, which buys concurrency, which buys throughput.max_num_seqs: the number of sequences the scheduler will run concurrently. Online you cap it to protect time-per-output-token. Offline, raise it until KV cache preemption starts thrashing, then back off one step.- Quantization: FP8 weights on Hopper or newer typically give 1.3-1.8x throughput with negligible quality change on extraction and classification tasks. Validate on a 500-row sample against the BF16 baseline, not on vibes.
- Prefix caching: if every prompt shares a 1,200-token system prompt and few-shot block, prefix caching eliminates that prefill for every row after the first. On template-heavy backfills this alone can halve the job.
- Speculative decoding: skip it. It trades throughput for latency, and you have no latency requirement.
One more lever specific to batch: shrink the output. Batch jobs are usually decode-bound, and decode cost is linear in tokens generated. Asking for strict JSON with short field values, capping max_tokens honestly, and disabling chain-of-thought where it does not change accuracy often cuts the bill more than any kernel-level optimization.
Rule three: assume you will be preempted
A 20-hour job on an on-demand H100 is expensive. The same job on spot or preemptible capacity costs 60-80% less, with the catch that the instance can disappear with 30 seconds of warning. That trade is almost always worth taking for offline work — if the job is restartable.
Restartability means two things: shard the work, and commit results incrementally.
import os, json, pathlib
OUT = pathlib.Path(os.environ["OUT_DIR"])
shard_id = int(os.environ["SHARD_ID"])
done_path = OUT / f"shard-{shard_id:05d}.jsonl"
# Resume: skip IDs already written
done = set()
if done_path.exists():
with done_path.open() as f:
done = {json.loads(line)["id"] for line in f}
todo = [d for d in shard_docs if d["id"] not in done]
with done_path.open("a") as f:
for chunk in chunks(todo, 512):
for doc, result in run_batch(chunk):
f.write(json.dumps({"id": doc["id"], "result": result}) + "\n")
f.flush()
os.fsync(f.fileno())
Append-only JSONL (or Parquet part files) keyed by a stable source ID gives you idempotent, exactly-once-at-the-row-level semantics: a rerun of a dead shard re-does at most one chunk. fsync after each chunk matters — buffered writes die with the instance.
Pick a chunk size so that a chunk takes 30-120 seconds. Smaller and the flush overhead and progress bookkeeping dominate; larger and preemption throws away real work.
Three more preemption habits worth building in:
- Trap the termination signal. Cloud preemption sends
SIGTERM; catch it, flush, and exit non-zero so the orchestrator reschedules the shard. - Never write to the final destination directly. Write part files, then do one atomic publish (a manifest row, a table swap, a prefix rename) once all shards report complete.
- Make the model artifact local and pinned. Pre-bake weights into the image or sync them to local NVMe on boot. A thundering herd of 64 spot workers pulling 16 GB of weights from object storage at once is a slow, costly way to start a job.
Rule four: scale out with embarrassingly parallel shards
Batch inference is embarrassingly parallel at the row level, so you almost never need distributed training machinery. The simplest architecture that works:
- A manifest of input shards (object storage paths or ID ranges), written once.
- N workers, each claiming shards by index (
SHARD_IDfrom a Kubernetes indexed Job, a Slurm array task, or Ray tasks). - A completion marker per shard, so re-runs skip finished work.
apiVersion: batch/v1
kind: Job
spec:
completions: 64
parallelism: 16
completionMode: Indexed
backoffLimit: 200 # spot deaths are expected, not exceptional
template:
spec:
restartPolicy: OnFailure
containers:
- name: infer
image: registry.internal/batch-infer:2026-03-11
resources:
limits: { nvidia.com/gpu: 1 }
Only reach for a heavier framework (Ray Data, Spark with a GPU UDF, or a managed batch service) when you genuinely need pipelined CPU preprocessing feeding multiple GPU actors — for example, PDF rendering or video decoding that would otherwise starve the GPU. In that case the useful pattern is a CPU-heavy stage with high parallelism feeding a small pool of GPU actors through a bounded queue, which keeps GPU utilization above 90% without an unbounded memory footprint.
For models too large for one GPU, prefer data parallelism (one full model copy per GPU, different shards) over tensor parallelism. Tensor parallelism adds collective communication on every token; replication adds none. Only shard the model when it does not fit.
Rule five: measure cost per million rows, not GPU hours
The number to track in a batch pipeline is cost per million rows (or per million tokens), broken out by stage. Instrument four things and you can usually find a 2x within an afternoon:
- GPU utilization and SM occupancy during steady state (
nvidia-smi dmon, or the PyTorch profiler on one worker for a 60-second window). - Rows per second per GPU, steady state, excluding startup.
- Fraction of wall clock spent on model load, data load and publish versus actual inference.
- Padding waste: total non-pad tokens divided by total tokens processed.
The usual findings, in rough order of frequency: the GPU sits idle waiting on a single-threaded CPU preprocessing step; half the processed tokens are padding; the job reloads weights from object storage on every shard; outputs are three times longer than anyone needs; and the cluster is running on-demand instances for work with a 12-hour deadline.
A short checklist
- Sort or bucket by length; keep the permutation to realign outputs.
torch.inference_mode(), bfloat16, pinned memory,num_workershigh enough that the GPU never waits.torch.compile(dynamic=True)with padded bucket sizes to limit recompiles.- For LLMs: offline engine API, high
gpu_memory_utilization, largemax_num_seqs, prefix caching, FP8 after a sampled quality check, no speculative decoding. - Shard the work; append results with a stable ID;
fsync; resume by skipping written IDs. - Run on spot with a
SIGTERMhandler and a generousbackoffLimit. - Publish atomically at the end; never partially overwrite the destination table.
- Report cost per million rows, before and after.
Offline batch inference is the least glamorous part of a PyTorch deployment and frequently the largest line item on the GPU invoice. It is also the easiest place to find large, safe wins, because the constraint that makes online optimization hard — someone is waiting — simply is not there.
If you are running large-scale scoring, embedding or extraction backfills and the bill or the runtime has gotten out of hand, get in touch. Reworking a batch pipeline is usually a short, well-bounded engagement with a number attached to the outcome.