Structured Outputs That Never Fail to Parse: Constrained Decoding in PyTorch

Ask an open model to "return JSON" and it will, roughly, most of the time. Then at 3am a stray markdown fence, a trailing comma, or a hallucinated field name takes down the downstream parser. Retry loops paper over it at the cost of latency and tokens, and they never make the failure rate zero.

Constrained decoding removes the problem at its source. Instead of hoping the model emits valid JSON, you mask the sampler at every step so that only tokens which can still lead to a valid document have non-zero probability. Invalid output becomes structurally impossible, not merely unlikely. This tutorial covers how the masking works, how to turn it on in vLLM and in Hugging Face transformers, what it costs, and where it quietly damages quality.

How constrained decoding works

Every decoding step produces a logits vector over the vocabulary. Constrained decoding inserts one operation before sampling:

1. Keep a state machine describing the set of legal strings — a JSON Schema, a regex, or a context-free grammar compiled to a finite-state or pushdown automaton.
2. At each step, ask the automaton which vocabulary tokens can legally come next from the current state.
3. Set the logits of every other token to -inf.
4. Sample normally, then advance the automaton with the chosen token.

The expensive part is step 2. Naively you would test all ~128k vocabulary entries against the grammar at every step. Modern libraries (Outlines, XGrammar, llguidance) precompute an index mapping automaton state → allowed token bitmask, so the runtime cost drops to a bitmask lookup and an add — typically well under 1% of step time once the grammar is compiled. Compilation itself takes anywhere from milliseconds to a few seconds, so cache compiled grammars by schema hash and never compile inside a request path.

One subtlety that catches people: the automaton is defined over characters, but sampling happens over tokens, and a single BPE token can straddle a grammar boundary. Good implementations handle this with token healing and by allowing any token whose character expansion is a valid prefix. Hand-rolled implementations usually get it wrong and produce false rejections on exactly the strings you care about.

The easy path: vLLM structured outputs

If you serve with vLLM, you already have this. Server-side, via the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

schema = {
    "type": "object",
    "properties": {
        "vendor":      {"type": "string"},
        "invoice_no":  {"type": "string"},
        "total_cents": {"type": "integer", "minimum": 0},
        "currency":    {"type": "string", "enum": ["USD", "EUR", "GBP"]},
        "line_items": {
            "type": "array",
            "maxItems": 50,
            "items": {
                "type": "object",
                "properties": {
                    "description":  {"type": "string"},
                    "amount_cents": {"type": "integer"},
                },
                "required": ["description", "amount_cents"],
                "additionalProperties": False,
            },
        },
    },
    "required": ["vendor", "invoice_no", "total_cents", "currency", "line_items"],
    "additionalProperties": False,
}

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": f"Extract the invoice fields:\n\n{doc}"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "invoice", "schema": schema, "strict": True},
    },
    temperature=0,
)

Offline, the same thing through SamplingParams:

from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
params = SamplingParams(
    temperature=0,
    max_tokens=1024,
    guided_decoding=GuidedDecodingParams(json=schema),
)
outs = llm.generate(prompts, params)

GuidedDecodingParams also takes regex=, choice= and grammar= (EBNF) instead of json=. Use choice=["approve", "deny", "escalate"] for classification — it is dramatically more reliable than parsing free text, and it costs a handful of tokens.

Exact keyword spellings drift between vLLM releases; pin your version and check the flags for the tag you deploy. The concepts are stable, the spelling is not.

Pydantic as the source of truth

Write the schema once, in Python, and let it drive both the constraint and the parse:

from pydantic import BaseModel, Field
from typing import Literal

class LineItem(BaseModel):
    description: str
    amount_cents: int

class Invoice(BaseModel):
    vendor: str
    invoice_no: str
    total_cents: int = Field(ge=0)
    currency: Literal["USD", "EUR", "GBP"]
    line_items: list[LineItem] = Field(max_length=50)

schema = Invoice.model_json_schema()
# ... generate with guided_decoding=GuidedDecodingParams(json=schema) ...
invoice = Invoice.model_validate_json(raw_output)   # cannot fail on structure

This is the pattern we deploy for clients: one Pydantic model is the API contract, the decoding constraint, the validation layer, and the thing your tests assert against. When the contract changes, exactly one file changes.

Local PyTorch, no server

For a plain transformers loop — research, a notebook, a batch job — the same machinery is available as a logits processor:

import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B-Instruct"
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="cuda"),
    AutoTokenizer.from_pretrained(model_id),
)

result = model(prompt, Invoice)          # returns a validated Invoice

And if you want to see the mechanism rather than hide it, a constraint as a raw LogitsProcessor is only a few lines:

import torch
from transformers import LogitsProcessor

class AutomatonMask(LogitsProcessor):
    """Mask logits to the tokens the compiled automaton permits from the current state."""

    def __init__(self, guide, tokenizer):
        self.guide, self.tok = guide, tokenizer
        self.state = guide.initial_state

    def __call__(self, input_ids, scores):
        allowed = self.guide.get_next_instruction(self.state).tokens  # LongTensor of ids
        mask = torch.full_like(scores, float("-inf"))
        mask[:, allowed] = 0.0
        return scores + mask

Advance self.state with each accepted token in a custom generate loop. In production, prefer the maintained library: batching, beam search, token healing and prefix caching all interact with this in non-obvious ways.

What it costs you

Constrained decoding is not free, and the costs are not the ones people expect.

Latency. Negligible per step once compiled — a bitmask lookup. Grammar compilation is the real cost, so cache it. A cold compile of a deeply nested schema inside a request can add hundreds of milliseconds.

Quality — the important one. Forcing structure can push the model off its natural reasoning path. Two well-documented effects:

• Field order matters. JSON objects are emitted in schema order, so a reasoning field placed after answer is generated after the answer is already committed — pure decoration. Put reasoning fields first if you want them to do any work.
• Over-tight constraints hurt. A regex that forces a date into \d{4}-\d{2}-\d{2} will happily produce a confidently wrong date rather than let the model express uncertainty. Prefer a nullable string plus validation over a constraint that makes "I don't know" inexpressible. Always leave an explicit null or "unknown" escape hatch for fields that may genuinely be absent.

Still keep the prompt. Constraints guarantee shape, never semantics. The schema cannot stop the model putting the shipping address in the vendor field. Describe the fields in the prompt and in schema description strings anyway — most implementations feed descriptions to the model.

Unbounded arrays. A schema with an unlimited array can drive the model into a repetition loop that only max_tokens stops. Set maxItems and maxLength on anything list- or string-shaped.

Testing it

Because structure is guaranteed, your test suite changes shape. Structural assertions become trivial and boring; move the effort to semantics.

• Golden set: 50-200 real documents with hand-labelled fields. Report per-field exact-match and F1, not one overall pass rate — an aggregate hides which field regressed.
• Adversarial inputs: empty documents, wrong language, two invoices in one file, scanned noise. Assert the model returns nulls rather than inventing values.
• Schema-change regression: re-run the golden set on every schema edit. Reordering fields alone can move accuracy several points.
• Escape-hatch rate: track how often the model uses null or "unknown". A sudden drop to zero usually means an over-tight constraint, not a smarter model.

When not to use it

Skip constrained decoding when the output is genuinely free prose; when you need the model to think at length before committing (use a two-call pattern — an unconstrained reasoning call, then a constrained extraction call over its own output); or when your provider's native tool-calling already enforces the schema server-side and is well tested for your model.

Everywhere else — extraction, classification, routing, agent tool arguments, structured evaluation — it is close to a free win. It replaces a probabilistic failure mode with a deterministic guarantee, and it deletes the retry loop, the JSON-repair helper, and the ticket that gets filed every few weeks when something fails to parse.

---

IntelliSensei builds and hardens PyTorch and LLM systems in production — extraction pipelines, serving stacks, evaluation harnesses. If you need structured output you can actually depend on, get in touch.