Serving PyTorch Without Python: torch.export and AOTInductor in Production

Almost every PyTorch serving stack we are asked to review has the same shape: a Python process, a torch.compile'd model, a web framework in front, and a warm-up loop that burns thirty to ninety seconds on every cold start while Inductor recompiles graphs that were already compiled yesterday. It works. It is also the reason those services cannot autoscale quickly, cannot be embedded in an existing C++ or Rust service, and drag a multi-gigabyte Python environment into every container image.

AOTInductor is the PyTorch-native answer. It takes an exported graph and compiles it ahead of time into a self-contained shared library that a C++ (or Python) runtime loads and calls directly. No tracing at startup, no guard checks, no Python on the hot path. This tutorial walks the whole path: export, compile, package, run, and the things that break.

When this is the right tool

AOTInductor is not a replacement for torch.compile in every service. Reach for it when at least one of these is true:

• Cold start matters. Serverless, spiky traffic, or a fleet that scales to zero. Compilation happens once in CI instead of once per replica.
• The host process is not Python. You have a C++ trading system, a Rust gateway, a Go service with cgo, or a game engine, and you want the model inside it rather than behind an RPC hop.
• You want a versioned model artifact. A single .pt2 file with a checksum that moves through your release pipeline, rather than "a checkpoint plus whatever pip freeze said that week".
• Latency variance is the problem. Recompilation on a new batch shape shows up as a multi-second p99 spike. Ahead-of-time compilation with declared dynamic shapes removes that class of stall entirely.

Stay with plain torch.compile when the model changes shape or control flow constantly, when you rely on Python-side pre/post-processing that is awkward to move into the graph, or when you are still iterating on the architecture. Export is a contract; contracts slow down experimentation.

Step 1: export the model

Everything starts with torch.export, the same front end used by ExecuTorch on-device. It produces a full-graph, ATen-level IR with no Python semantics left in it.

import torch
from torchvision.models import resnet50, ResNet50_Weights

model = resnet50(weights=ResNet50_Weights.DEFAULT).eval().cuda()
example = (torch.randn(8, 3, 224, 224, device="cuda"),)

batch = torch.export.Dim("batch", min=1, max=64)
ep = torch.export.export(
    model,
    example,
    dynamic_shapes={"x": {0: batch}},
)

Two decisions here matter more than the rest of the tutorial.

Declare your dynamic dimensions. Anything you do not declare is burned in as a constant. If you export with a batch of 8 and never declare batch, a request with 5 items fails at runtime. Declare the dimensions that genuinely vary — batch, sequence length — and give them realistic min/max bounds; the bounds feed kernel selection, and an absurd max produces conservative, slower code.

Export in eval mode with the exact preprocessing you serve. If normalization or tokenizer-adjacent tensor work lives in Python today, consider pulling it inside the exported module. Anything left outside the graph is code you must reimplement in C++ later, and reimplement identically, or you get silent accuracy drift.

Export fails loudly on data-dependent control flow (if x.sum() > 0:), on .item() calls, and on arbitrary Python objects in the signature. That noise is the point — it is exactly the code that would have caused a graph break and a slow path under torch.compile.

Step 2: compile ahead of time

package_path = torch._inductor.aoti_compile_and_package(
    ep,
    package_path="resnet50_h100.pt2",
    inductor_configs={
        "max_autotune": True,
        "triton.cudagraphs": True,
    },
)

max_autotune benchmarks candidate kernels and picks winners. It makes compilation slow — minutes, sometimes tens of minutes for a large transformer — and that is fine, because this runs in CI, not in your service. The output .pt2 is a zip containing the compiled shared object, the constants, and the metadata the runtime needs.

The artifact is specific to the GPU architecture and the PyTorch build used to produce it. An artifact compiled on an H100 with CUDA 12.6 is not guaranteed to load on an L4, and definitely not on a different major PyTorch version. Treat the target device as part of the build matrix: one artifact per (model version, architecture, runtime version). Bake that triple into the filename, not into a wiki page.

Step 3: run it

From Python, for testing and for the many services that are perfectly happy staying in Python:

import torch

runner = torch._inductor.aoti_load_package("resnet50_h100.pt2")
out = runner(torch.randn(5, 3, 224, 224, device="cuda"))

From C++, which is the reason most teams are here:

#include <torch/csrc/inductor/aoti_package/model_package_loader.h>

torch::inductor::AOTIModelPackageLoader loader("resnet50_h100.pt2");
std::vector<torch::Tensor> inputs = {
    torch::randn({5, 3, 224, 224}, at::kCUDA)};
std::vector<torch::Tensor> outputs = loader.run(inputs);

Link against libtorch, ship the .pt2 next to the binary, and the Python interpreter is gone from the serving path. In a container this typically takes an image from several gigabytes to a few hundred megabytes plus CUDA, and cold start from "warm-up loop" to "load a file".

Step 4: prove it before you commit

Two checks, both cheap, both routinely skipped.

Numerical parity. Run the eager model and the compiled artifact over a few thousand real inputs and compare. Use a tolerance you have justified (torch.testing.assert_close with explicit rtol/atol), and compare task metrics too, not just tensors — for a classifier, top-1 agreement; for an LLM, greedy-decode token match on a fixed prompt set. Autotuning can select kernels with slightly different accumulation order, and "slightly different" occasionally means a different argmax.

Shape coverage. Sweep every shape in your declared range, including min, max, and the awkward ones in between. A shape that only appears at 3 a.m. under a partial batch is a bad time to discover an unsupported specialization.

Put both in CI, gated on the artifact, and store the results next to the artifact. The whole value proposition of ahead-of-time compilation is that the thing you tested is byte-for-byte the thing you ship.

What we see go wrong

Silent specialization. The model exported fine, the tests passed at batch 8, and production sends batch 1 for the first request of the day. Always test the boundaries of every dynamic dim.

The dependency drift trap. Someone bumps PyTorch in the training image, the artifact is rebuilt automatically, and the C++ service still links last quarter's libtorch. Version the runtime and the artifact together and refuse to load on mismatch — the loader will tell you, but only if you surface the error rather than falling back silently.

Preprocessing divergence. The Python service resized with antialiasing; the C++ service did not. Accuracy drops two points and nobody suspects the resize filter for a week. Push preprocessing into the graph, or pin it in a shared library used by both paths.

Autotuning on the wrong hardware. Compiling on an A100 because that is what the CI runner has, then serving on L40S. You get correct results and mediocre kernels. Compile on the hardware you serve on.

Chasing AOTI when the bottleneck is elsewhere. If your p99 is dominated by tokenization, a data-loading stall or a network hop, a python-free runtime changes nothing. Profile first — we have talked more than one client out of an export project after an afternoon with the profiler.

Using it for LLM serving where a dedicated engine wins. For high-throughput autoregressive generation, continuous batching and paged attention matter far more than compile-time kernel selection; vLLM will beat a hand-rolled AOTI server on tokens per second. AOTI shines on encoders, vision models, rankers, embedding models and small custom nets — the high-QPS, fixed-shape, low-latency workloads.

A reasonable rollout

1. Pick one non-LLM model with stable inputs and painful cold starts.
2. Export it, fix whatever export complains about, and keep those fixes in the training code — they make the model better under torch.compile too.
3. Compile per target architecture in CI; publish .pt2 artifacts to the same registry as your containers.
4. Deploy behind a flag with the Python path still live; compare latency, p99 and task metrics for a week.
5. Only then remove Python from the serving image.

Done in that order the change is boring, which is what you want from a deployment mechanism. Done as a big-bang rewrite it produces a fast service that nobody trusts.

We do this work with client teams regularly — export surgery on models that were never written to be exportable, per-architecture build pipelines, parity harnesses and the C++ integration itself. See our PyTorch performance optimization and PyTorch cloud deployment services, or contact us with your model and latency target and we will tell you whether ahead-of-time compilation is worth it for you.