LLM Inference and Serving Optimization

Training a model is a one-off cost. Serving it is a cost you pay every second it is running. IntelliSensei engineers LLM inference for the three numbers that matter in production: latency, throughput and cost per token. We work in native PyTorch and the open serving ecosystem built on it, on your cloud or on your own GPUs.

Throughput and latency profiling

Every engagement starts with measurement. We profile your current deployment for time-to-first-token, tokens per second per GPU, p50/p95 latency under realistic concurrency, and GPU utilization. Most first-generation LLM deployments we see run below 30% utilization; the profile tells us where the rest went.

vLLM and SGLang deployment

For open-weight LLMs the production answer in 2026 is a purpose-built inference engine. We deploy and tune vLLM and SGLang, which use paged attention and continuous batching to serve many concurrent requests from one GPU and expose an OpenAI-compatible API so your application code does not change. We tune the knobs that matter (KV-cache budget, maximum concurrent sequences, prefix caching, speculative decoding where it pays) against your real traffic shape.

Quantized inference

Quantization is the single biggest cost lever. We apply INT8, FP8 and INT4 weight quantization using AWQ/GPTQ-class methods and torchao, validate accuracy on your evaluation set, and pick the lowest precision that holds quality. On current NVIDIA hardware FP8 typically halves memory and raises throughput with near-zero quality loss; INT4 lets a model that needed two GPUs fit on one.

torch.compile and CUDA-graph capture

For latency-critical paths, especially encoder models, rerankers and embedding services, we use torch.compile and CUDA-graph capture to remove Python and kernel-launch overhead. This is often a 1.5-3x latency improvement on small-batch inference with no change to the model.

Multi-GPU serving and autoscaling

Larger models are served with tensor parallelism across GPUs; bursty traffic is handled with autoscaling on AWS, GCP or Azure. We design the scaling policy around queue depth and token throughput rather than CPU, set up warm pools so a scale-out does not mean a cold model load, and put the whole thing behind standard observability (Prometheus metrics, traces, per-request token accounting).

Cost-per-token engineering and capacity planning

We give you a cost model: dollars per million input and output tokens at your traffic levels, for each candidate configuration (model size, precision, GPU type, batching policy). That model turns "how many GPUs do we need?" into an arithmetic question and tells you when reserved or owned hardware beats on-demand.

TorchServe migration

The pytorch/serve repository was archived in August 2025: no further updates, bug fixes or security patches. If you are running TorchServe in production you are running unmaintained software. We migrate TorchServe deployments to supported platforms: NVIDIA Triton Inference Server, LitServe, ONNX Runtime, or vLLM for LLM workloads, with golden-output parity tests so the new endpoint behaves exactly like the old one before traffic moves. Our tutorial Migrating off TorchServe walks through the decision tree.

Deliverables

A serving configuration checked into your repository, a load-test harness with before/after numbers, a cost model, and dashboards. Where we quantize, you get the accuracy comparison alongside the speed-up.

Paying too much per token, or stuck on TorchServe? Contact us and we will profile your current deployment as the first step.

Back to services