Monitoring PyTorch Models in Production: Drift, Silent Regressions and Canary Evals

Every team we work with has a plan for getting a model into production and almost none have a plan for the six months after. The model ships, the eval numbers get pasted into a slide, and then the only signal anyone watches is p99 latency and the error rate. Quality decay is invisible to both: a retrieval index that has drifted stale, a prompt template change, a tokenizer swap, a vendor model version bump, or simply a shift in what users ask about will all degrade answers while every dashboard stays green.

This tutorial is the monitoring stack we install on PyTorch inference services: what to log, which drift statistics are worth computing, how to catch silent quality regressions with canary evals, and how to wire it all into Prometheus without turning your serving path into a data pipeline.

The four layers worth monitoring

Treat them as separate budgets with separate alerts. Conflating them is why most ML monitoring gets ignored.

Layer	Question	Example signals
Service health	Is it up and fast?	QPS, TPOT, TTFT, GPU utilization, OOM/CUDA errors, queue depth
Input drift	Are we seeing different data?	Embedding distribution shift, prompt length, language mix, null/OOV rates
Output drift	Is the model behaving differently?	Score distributions, refusal rate, output length, parse-failure rate, tool-call rate
Ground truth	Is it still right?	Delayed labels, canary eval scores, human review, user thumbs-down

Only the last layer measures accuracy, and it is always the latest to arrive. The first three are early-warning proxies you can compute in real time. The whole point of the design is to alert on cheap proxies and confirm with expensive truth.

Step 1: log the right things once, at the edge

Log a compact record per request, asynchronously, and never on the critical path.

import asyncio

log_queue: asyncio.Queue = asyncio.Queue(maxsize=10_000)

async def log_writer(sink):
    batch = []
    while True:
        try:
            batch.append(await asyncio.wait_for(log_queue.get(), timeout=2.0))
        except asyncio.TimeoutError:
            pass
        if batch:
            await sink.write_many(batch)   # object store / warehouse
            batch = []

def record(**fields):
    try:
        log_queue.put_nowait(fields)
    except asyncio.QueueFull:
        pass                               # drop logs, never drop traffic

The per-request payload we standardize on:

• request_id, timestamp, model_name, model_version, prompt_template_version
• input summary: token count, language, truncation flag, retrieved-doc ids (for RAG)
• a fixed-dimension input embedding (see step 2) or a quantized digest of it
• output summary: token count, finish reason, mean token logprob, parse success, tool calls
• latency split: queue, prefill, decode
• a feedback slot, filled in later by a join on request_id

Two rules save real pain. Version everything that can change the output — weights, prompt template, retriever index, tokenizer — because most "drift" incidents turn out to be undeclared deploys. And sample heavy payloads: 100% of summaries, 1-5% of full inputs and outputs, with a forced 100% sample for any request the user flagged.

Step 2: measure input drift in embedding space

For tabular features, per-feature statistics are fine. For text, images and audio, compute drift on embeddings instead: it gives you one number for arbitrarily messy inputs.

Fit a reference from a known-good window, then score live batches against it.

import torch

class EmbeddingDriftMonitor:
    """Population-stability-style drift on projected embeddings."""

    def __init__(self, reference: torch.Tensor, n_proj: int = 16, bins: int = 10):
        ref = reference.float()
        g = torch.Generator().manual_seed(0)
        # Fixed random projections: cheap, stable, no PCA refit drift.
        self.P = torch.randn(ref.shape[1], n_proj, generator=g)
        self.P /= self.P.norm(dim=0, keepdim=True)
        proj = ref @ self.P
        qs = torch.linspace(0, 1, bins + 1)[1:-1]
        self.edges = torch.quantile(proj, qs, dim=0)          # [bins-1, n_proj]
        self.ref_frac = self._hist(proj)

    def _hist(self, proj):
        idx = torch.searchsorted(
            self.edges.T.contiguous(), proj.T.contiguous(), right=True
        )                                                     # [n_proj, N]
        bins = self.edges.shape[0] + 1
        counts = torch.zeros(proj.shape[1], bins)
        counts.scatter_add_(1, idx, torch.ones_like(idx, dtype=torch.float))
        return (counts + 1e-6) / counts.sum(1, keepdim=True)

    def psi(self, live: torch.Tensor) -> torch.Tensor:
        """Per-projection PSI. >0.1 investigate, >0.25 act."""
        p = self._hist(live.float() @ self.P)
        return ((p - self.ref_frac) * (p / self.ref_frac).log()).sum(1)

    def score(self, live: torch.Tensor) -> float:
        return self.psi(live).max().item()

Notes from deploying this:

• Random projections beat PCA for monitoring. A refit PCA basis makes yesterday's numbers incomparable to today's; a seeded random projection is frozen forever.
• Use the same embedding model you serve with, at the same precision. A drift monitor with a different encoder measures the encoder, not the traffic.
• Score in windows of at least a few hundred requests. PSI on 20 samples is noise.
• Alert on the max across projections but log all of them; the projection that fires tells you roughly which direction traffic moved.
• Reset the reference window deliberately, with a changelog entry. Auto-rebaselining hides slow decay, which is the failure mode you most want to catch.

Drift is a hypothesis, not an incident. It means "your eval set may no longer describe production" — useful, and not the same as "quality dropped."

Step 3: output-side guards that correlate with quality

For generative systems these cheap signals catch most real regressions:

• Parse-failure rate for structured outputs — the single best canary for a broken prompt or an unannounced model swap.
• Refusal rate, detected with a small classifier rather than string matching.
• Output length distribution. A sudden collapse in mean length is a classic sign of a truncated template or a bad stop-token config.
• Mean token logprob of the generated text. A drop means the model is less confident in its own output; combined with input drift it is a strong out-of-distribution signal.
• Retrieval grounding: similarity between the answer and its retrieved context, plus the fraction of answers whose top retrieved doc scored below threshold. A cheap proxy for hallucination risk in RAG.
• For classifiers, mean max-softmax and predicted-class mix. A class whose share doubles overnight is either a real world event or a broken feature.

@torch.inference_mode()
def output_signals(logprobs: torch.Tensor, text: str, ctx_emb, ans_emb):
    return {
        "mean_logprob": logprobs.mean().item(),
        "low_conf_frac": (logprobs < -3.0).float().mean().item(),
        "n_tokens": logprobs.numel(),
        "parse_ok": try_parse(text),
        "grounding": torch.cosine_similarity(ans_emb, ctx_emb, dim=-1).max().item(),
    }

Export them as histograms, not averages. An average hides a bimodal failure where 5% of traffic is completely broken.

from prometheus_client import Histogram, Counter

TPOT = Histogram("tpot_seconds", "time per output token", ["model_version"])
GROUNDING = Histogram("rag_grounding", "answer-context similarity",
                      buckets=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])
PARSE_FAIL = Counter("parse_failures_total", "structured output parse failures",
                     ["model_version", "schema"])

Step 4: canary evals on a schedule

Proxies tell you something changed. A canary eval tells you whether quality moved. Run a small fixed suite continuously against the live endpoint — not against a local copy of the weights, because half of all regressions live in the serving config.

• 100-300 cases with stable, checkable answers, drawn from real traffic and then frozen.
• Run on every deploy and on a cron: hourly for high-stakes systems, daily otherwise.
• Fix seeds and use temperature 0 where you can; where you cannot, run n samples and compare distributions.
• Track score per slice (customer tier, language, document type, query length). Aggregates hide the regression that only hits one segment.
• Gate deploys on it. A canary eval that cannot block a rollout will be ignored within a month.

Pair it with a replay harness: take yesterday's 1% sampled production inputs, run them through the candidate build, and diff outputs against what production actually returned. You are not scoring correctness here, just measuring churn. A 40% output-change rate on a "minor" dependency bump is exactly the thing you want to see before promotion.

Step 5: alerts people will not mute

Alert quality is the single biggest determinant of whether monitoring survives contact with an on-call rotation.

• Page only on things that need a human within minutes: error rate, latency SLO burn, parse-failure spike, throughput collapse.
• Ticket, do not page, on drift and canary-score dips. They need investigation, not a 3am response.
• Require two consecutive windows before firing. Single-window ML alerts are almost always noise.
• Every alert carries a version stamp and links to the sampled examples that triggered it. An alert you cannot immediately inspect examples for is an alert you will mute.
• Annotate dashboards with deploys. Most "mystery drift" resolves on sight once the deploy line is visible on the chart.

A minimal rollout order

Starting from nothing, this is the order that gives the most value per week of effort:

1. Async request logging with full version stamping. Nothing else works without it.
2. Prometheus histograms for latency, output length, parse failures and finish reasons.
3. A frozen canary eval suite wired into the deploy gate.
4. An embedding drift monitor on inputs, with reference windows checked into version control.
5. A feedback join, so thumbs-down requests are retrievable by request_id.
6. A weekly review where a human reads 20 sampled low-confidence outputs.

Step 6 sounds unserious and is consistently the highest-yield item on the list.

Closing

Monitoring an ML system is not a dashboard, it is a loop: cheap real-time proxies raise a hypothesis, a canary eval or a labeled sample confirms it, and the confirmed cases become new eval cases and new training data. Build the loop and quality decay becomes a ticket with evidence attached instead of a customer escalation six weeks late.

If you want help instrumenting an existing PyTorch service — drift monitoring, canary eval suites, deploy gating, or a retraining trigger you can defend to an auditor — get in touch.