Most PyTorch deployment writing stops at the server. But a growing share of the models our clients build in 2026 never touch a datacentre GPU at inference time: they run on phones, wearables, cameras, handheld scanners and vehicle head units, because the latency budget is 30 ms, the device is sometimes offline, or the data legally cannot leave the handset.
ExecuTorch is the PyTorch-native answer to that. It is the successor to the old TorchScript-based PyTorch Mobile path, and it works very differently: instead of shipping a Python-derived interpreter, you export an ahead-of-time graph, lower it to hardware-specific backends, and load a self-contained .pte file from a small C++ runtime. This tutorial walks the whole path with a realistic image model, then covers the failure modes that only appear once real devices are involved.
The mental model
There are three stages, and keeping them distinct saves a great deal of debugging:
- Export.
torch.exporttraces yournn.Moduleinto an ATen graph with an explicit set of input shapes and constraints. This is a strict, full-graph capture: no graph breaks allowed, unliketorch.compile. - Transform. Quantize, apply memory-planning and partition the graph across delegates (XNNPACK for CPU, Core ML or MPS on Apple hardware, Vulkan for mobile GPUs, plus vendor NPU backends such as Qualcomm QNN and MediaTek Neuropilot).
- Emit. Serialize to a
.ptebinary and load it from the ExecuTorch runtime in a Swift, Kotlin or C++ app.
Stages 1 and 2 happen on your laptop or in CI. Stage 3 happens on the device. Everything that goes wrong in production is either an export-time constraint you did not declare or a delegate that silently declined to take part of your graph.
Step 0: install
pip install executorch torch torchvision torchao
ExecuTorch and PyTorch versions are coupled. Pin both in requirements.txt and rebuild your device runtime whenever you bump them; a .pte produced by a newer exporter than the on-device runtime will fail to load with an unhelpful schema error.
Step 1: export the model
Start from an ordinary eval-mode module and an example input.
import torch
import torchvision
model = torchvision.models.mobilenet_v3_small(weights="DEFAULT").eval()
example_inputs = (torch.randn(1, 3, 224, 224),)
exported = torch.export.export(model, example_inputs)
print(exported.graph_module.code[:500])
If this fails, it fails here and not later — which is the point. The usual causes are data-dependent control flow (if x.sum() > 0:), Python side effects, .item() calls, and dictionaries of tensors with varying keys. Fixes, in order of preference: remove the dynamism, replace it with torch.cond / torch.ops.higher_order constructs, or split the model so the dynamic part stays in application code.
Declaring dynamic shapes
A fixed 224x224 batch-of-one export is fine for a classifier and useless for a text or audio model. Declare the axes that vary:
from torch.export import Dim
seq = Dim("seq", min=1, max=512)
exported = torch.export.export(
text_model,
(torch.randint(0, 32000, (1, 128)),),
dynamic_shapes={"input_ids": {1: seq}},
)
Be explicit about min and max. An unbounded dimension forces the runtime to plan for the worst case, and on a device with 4 GB of RAM the worst case is what gets you killed by the OS. Under-declaring is worse: passing a length-600 input to a graph exported with max=512 is a runtime error on the user's phone, not on your laptop.
Step 2: quantize for the device
CPU and NPU backends want INT8. ExecuTorch uses the PT2 export quantization flow, with a backend-specific quantizer that knows which patterns the target can actually fuse.
from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import (
XNNPACKQuantizer, get_symmetric_quantization_config,
)
quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config())
graph = torch.export.export_for_training(model, example_inputs).module()
prepared = prepare_pt2e(graph, quantizer)
for batch in calibration_loader: # 100-500 representative samples
prepared(batch)
quantized = convert_pt2e(prepared)
exported = torch.export.export(quantized, example_inputs)
The calibration set matters more than the algorithm. Use real inputs from the deployment distribution — actual phone-camera frames, not clean validation-set JPEGs. A model calibrated on studio images and deployed against low-light handheld photos will lose several points of accuracy that no amount of quantizer tuning recovers.
For LLM-shaped models on device, weight-only 4-bit is the usual choice instead; torchao's Int4WeightOnlyConfig and ExecuTorch's LLM export recipes cover that path, and the same precision reasoning we set out in BF16, FP8, INT8: choosing precision applies.
Step 3: lower to a delegate and emit the .pte
from executorch.exir import to_edge_transform_and_lower
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
program = to_edge_transform_and_lower(
exported,
partitioner=[XnnpackPartitioner()],
).to_executorch()
with open("mobilenet_v3_int8.pte", "wb") as f:
f.write(program.buffer)
Choose the partitioner by target:
| Target | Delegate | Notes |
|---|---|---|
| Any ARM or x86 CPU | XNNPACK | The safe default; broad operator coverage, INT8 and FP32 |
| iPhone / iPad / Mac | Core ML | Reaches the Apple Neural Engine; coverage varies by iOS version |
| Apple GPU | MPS | Useful when Core ML declines large parts of the graph |
| Android GPU | Vulkan | Good for vision models; check per-vendor driver quirks |
| Qualcomm NPU | QNN | Best power efficiency on Snapdragon; strictest operator set |
| Microcontroller | portable / ARM Ethos-U | Kilobyte-scale runtime, tiny operator set |
Always inspect what was actually delegated. Partitioners fail open: unsupported subgraphs quietly fall back to the portable CPU kernels, which are correct and slow.
from executorch.exir.backend.utils import print_delegated_graph
print_delegated_graph(program.exported_program().graph_module)
If you see your convolutions running as portable ops, that is your missing 5x.
Step 4: validate before it ships
Run the emitted program through the Python runtime and diff against eager PyTorch on the same inputs:
from executorch.runtime import Runtime
runtime = Runtime.get()
method = runtime.load_program("mobilenet_v3_int8.pte").load_method("forward")
out_et = method.execute([example_inputs[0]])[0]
out_eager = model(*example_inputs)
print((out_et - out_eager).abs().max())
For an INT8 model, expect visible numerical difference — that is quantization, not a bug. The check that matters is task accuracy on a held-out set, computed from the .pte, not from the FP32 model. Make it a CI job: export, lower, evaluate, and fail the build if accuracy drops more than an agreed threshold (we usually set 1% relative for classifiers, tighter for anything safety-adjacent).
Step 5: load it in the app
On Android, add the ExecuTorch AAR and load from assets:
val module = Module.load(assetFilePath(this, "mobilenet_v3_int8.pte"))
val output = module.forward(EValue.from(inputTensor))[0].toTensor()
On iOS, add the ExecuTorch Swift package and use the equivalent Module API. In both cases the runtime is on the order of a few hundred kilobytes, links statically, and has no Python dependency — which is the whole reason the export discipline in step 1 exists.
Failure modes we see in the field
Export succeeds, device fails. Nearly always an undeclared dynamic dimension. Export with the shape ranges your app can actually produce and fuzz them in CI.
Great benchmark, poor field latency. Phones thermally throttle. A model that runs in 18 ms for three frames runs in 45 ms after two minutes of sustained camera use. Benchmark with a sustained workload, on a warm device, on the cheapest handset you support.
Accuracy fine in CI, bad in the wild. Preprocessing drift. The resize filter, colour space and normalization constants in the mobile code must match training exactly; put preprocessing inside the exported graph where you can, so it cannot diverge.
Model size blocks the app store. Ship the .pte as a post-install download, not in the bundle, once you pass a few tens of megabytes. Version it against the app so an old binary never loads a newer program.
Everything is slow on one vendor's phones. Delegate fallback, a driver bug, or an NPU that only accepts a narrow operator set. Keep an XNNPACK build as the tested fallback and select the delegate at runtime by device tier.
When on-device is the wrong answer
On-device inference is not free. You take on per-vendor testing, staged model rollouts, and a much slower iteration loop than a server deployment where a new model is one container push. If your latency budget tolerates a round trip, your users are online, and the data can leave the device, serve it centrally — vLLM or a compiled server model will get you there faster and be easier to update. Go on-device when offline operation, privacy, per-inference cost at very high volume, or a hard sub-50 ms budget makes the trade worth it.
We build and ship these pipelines end to end — export, quantization, delegate selection, device benchmarking and the CI harness that keeps them honest. See our edge and on-device AI with ExecuTorch and PyTorch performance optimization services, or contact us with your model and target device and we will tell you what is realistic.