+1 (726) 227-4060
IntelliSensei

Migrating off TorchServe: A Practical Guide

On August 7, 2025 the pytorch/serve repository was archived. The notice is blunt: TorchServe is no longer actively maintained, and there are no planned updates, bug fixes, new features or security patches. If you are serving models with it, the question is not whether to migrate but to what, and how to do it without an outage. This guide is the checklist we use with clients.

1. Know what you are running

TorchServe deployments accumulate state in three places, and all three need inventorying before you choose a target:

  • Model archives (.mar files). Each bundles serialized weights, a handler, and optional extra files. List them with ls model_store/ and curl localhost:8081/models.
  • Handlers. The Python class per model that implements preprocess, inference and postprocess. This is where the business logic hides: image decoding, tokenization, thresholding, label maps. Read every one.
  • Configuration. config.properties (workers per model, batch size, max_batch_delay, GPU assignment) and any custom metrics or logging. Batching settings in particular need to be carried over deliberately.

Write down, per model: input format, output format, batch configuration, GPU/CPU placement, and the latency SLO it is held to. This table drives every decision below.

2. Choose the target

There is no single replacement for TorchServe; there are four credible ones, and the right choice depends on the workload.

NVIDIA Triton Inference Server. The right answer for high-throughput GPU serving and for mixed fleets (PyTorch, ONNX, TensorRT in one server). Dynamic batching, model ensembles, concurrent model execution, and first-class metrics. The cost is operational complexity and a model-repository layout to learn. Choose it when you have many models on shared GPUs or a serious throughput requirement. Docs.

LitServe. A lightweight, Python-first serving framework built on FastAPI with batching, streaming and multi-GPU workers built in. It is closest in spirit to TorchServe's handler model, which makes migration mechanical: your preprocess/inference/postprocess map almost one-to-one onto decode_request/predict/encode_response. Choose it when your handlers are Python-heavy and you want to keep them. Repo.

ONNX Runtime. Export to ONNX and serve with ORT, optionally behind Triton. Best when you need CPU inference, cross-language clients (C#, Java, C++), or the model is a standard architecture that exports cleanly. Choose it when portability matters more than PyTorch-specific features.

Plain FastAPI + torch.export. For one or two models at moderate traffic, a FastAPI app that loads an exported program and runs it is perfectly adequate and has the fewest moving parts. Choose it when the team is small and the workload is simple.

vLLM is the answer if the model is an LLM; TorchServe was never the right tool for that and nothing below applies. See Serving an LLM with vLLM.

A quick decision rule: LLM → vLLM; many models or high throughput on GPU → Triton; Python-heavy handlers → LitServe; CPU or polyglot clients → ONNX Runtime; small and simple → FastAPI.

3. A worked migration: image classification to LitServe

Take a typical TorchServe image classifier. The handler decodes a JPEG, normalizes, runs a ResNet, and returns top-5 labels. Here is the same endpoint on LitServe.

pip install litserve torch torchvision pillow

# server.py
import base64, io, json
import torch
import torchvision
from PIL import Image
import litserve as ls


class ClassifierAPI(ls.LitAPI):
    def setup(self, device):
        weights = torchvision.models.ResNet50_Weights.DEFAULT
        self.model = torchvision.models.resnet50(weights=weights).to(device).eval()
        self.model = torch.compile(self.model)
        self.transform = weights.transforms()
        self.labels = weights.meta["categories"]
        self.device = device

    def decode_request(self, request):
        # Same wire format the TorchServe handler accepted: base64 JPEG.
        img = Image.open(io.BytesIO(base64.b64decode(request["image"]))).convert("RGB")
        return self.transform(img)

    def batch(self, inputs):
        return torch.stack(inputs).to(self.device)

    @torch.no_grad()
    def predict(self, x):
        return torch.softmax(self.model(x), dim=1)

    def unbatch(self, output):
        return list(output)

    def encode_response(self, probs):
        top = torch.topk(probs, 5)
        return {self.labels[i]: float(p) for p, i in zip(top.values, top.indices)}


if __name__ == "__main__":
    api = ClassifierAPI(max_batch_size=16, batch_timeout=0.01)
    ls.LitServer(api, accelerator="auto", workers_per_device=2).run(port=8000)

max_batch_size and batch_timeout replace TorchServe's batchSize and maxBatchDelay; workers_per_device replaces minWorkers. The request and response formats are kept identical so clients do not change during cutover. Run it with python server.py.

For Triton the equivalent work is exporting the model (torch.export → torch.aoti_compile_and_package, or ONNX) into a model repository with a config.pbtxt declaring inputs, outputs and dynamic batching, and moving the pre/post-processing into either the client or a Python backend model in an ensemble.

4. Parity testing

Do not cut over on the strength of "it works on my laptop." Prove parity with the old server on golden data:

  1. Collect golden inputs and outputs. Replay 500-1,000 real requests (sampled from production logs, with consent and PII handling) against the existing TorchServe endpoint and store request/response pairs.
  2. Replay against the new endpoint and compare. For classification compare top-k labels and assert probabilities agree within a tolerance (1e-3 is reasonable across a dtype or compile change; anything larger needs explaining).
  3. Compare latency distributions, not means: p50, p95, p99 at the same concurrency, using the same load tool (hey, k6, or Locust).
  4. Compare failure behaviour. Send malformed inputs, oversized images and empty bodies to both and check the new server returns equivalent status codes.

Keep the parity script in the repository; it becomes the regression test for every future model update.

5. Cutover checklist

  • New server deployed alongside the old one, on separate ports or hosts.
  • Metrics and logs flowing from the new server to the same dashboards (LitServe and Triton both expose Prometheus metrics).
  • Parity report signed off.
  • Traffic shifted with a weighted route (5% → 25% → 50% → 100%) at the load balancer, with automatic rollback on error-rate or p99 regression.
  • TorchServe left running but idle for one release cycle, then decommissioned.
  • torchserve and torch-model-archiver removed from requirements and container images, so the unmaintained packages do not linger.
  • Runbook updated: how to deploy a new model version, how to roll back.

Common migration problems

Handler logic that depended on TorchServe context (ctx.system_properties, manifest access) needs rewriting; there is no equivalent, and that is usually an improvement. Custom metrics emitted through TorchServe's metrics API need re-emitting through Prometheus client libraries. Models saved with torch.jit (TorchScript) still load in current PyTorch but TorchScript is in maintenance mode; take the opportunity to move to torch.export or plain state_dict loading. Multi-model .mar files that shared a handler should become one service per model, or a Triton ensemble.

Do not wait

Unpatched serving software in the request path is a security finding waiting to happen, and every month on TorchServe makes the eventual migration harder as the ecosystem moves on. If you would like help, our LLM inference and serving optimization practice includes TorchServe migration with parity testing and staged cutover. Contact us to talk through your deployment.