Serving Reasoning Models: Thinking Budgets, Routing and the Real Cost of Test-Time Compute

Reasoning models changed the shape of an inference bill. A classic instruct model answers a support question in 120 tokens; a reasoning model may burn 2,000 hidden thinking tokens first and then emit the same 120. Quality on hard tasks goes up, sometimes dramatically. Latency and cost per request go up too — and, unlike a bigger model, the increase is variable: the same prompt can cost 300 tokens one day and 4,000 the next.

This tutorial is about getting that variance under control. How to cap thinking, how to route only the requests that deserve it, how thinking tokens interact with KV cache and batching, how to parse and store reasoning safely, and how to prove on your own evals that the extra tokens are buying accuracy rather than rambling.

We assume open-weight reasoning models (the DeepSeek-R1 distills, Qwen3 in thinking mode, gpt-oss, Magistral-style models) served with vLLM or SGLang on your own GPUs — the situation most of our clients are in when they call us.

1. What actually changes at serve time

A reasoning model is still a normal autoregressive transformer. Three practical differences:

• Output length distribution is long-tailed. Median output might be 400 tokens, p99 8,000. Mean-based capacity planning underestimates badly.
• Part of the output is not for the user. Thinking is delimited (<think>...</think> or a separate channel) and usually stripped before display, but you still pay for every token of it.
• The decode phase dominates. Prefill is unchanged; decode is 5-20x longer. Anything that helps decode — better batching, quantized KV, speculative decoding — matters more than it did.

The direct consequence for a serving cluster: per-request KV cache lifetime grows with the thinking length, so the number of concurrently resident sequences at a given memory budget drops. A cluster sized for 800-token answers can start preempting and swapping when the same traffic starts thinking.

2. Cap the thinking, per request

The first control is a hard ceiling. Never run a reasoning model with an unbounded max_tokens on a user-facing path.

Three levels of control, in order of bluntness:

A. Total token cap. Always set it. It protects the cluster but truncates mid-thought, which produces an unusable answer — the worst outcome per dollar.

B. Thinking budget with forced closure. Better: allow N thinking tokens, then inject the closing tag and let the model answer. vLLM exposes a reasoning parser for splitting the fields; the budget itself is easiest to enforce in your gateway with a two-phase call:

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-8B", max_model_len=32768)

THINK_BUDGET = 1024
ANSWER_BUDGET = 512

def bounded_reasoning(prompt: str) -> tuple[str, str, int]:
    # Phase 1: think, but only so far. Stop early if the model closes on its own.
    think = llm.generate(
        prompt,
        SamplingParams(max_tokens=THINK_BUDGET, temperature=0.6, top_p=0.95,
                       stop=["</think>"], include_stop_str_in_output=True),
    )[0].outputs[0]
    trace = think.text
    if not trace.rstrip().endswith("</think>"):
        # Budget exhausted: force closure so the model must commit to an answer.
        trace += "\n</think>\n"
    # Phase 2: answer, conditioned on however much thinking we allowed.
    ans = llm.generate(
        prompt + trace,
        SamplingParams(max_tokens=ANSWER_BUDGET, temperature=0.6, top_p=0.95),
    )[0].outputs[0]
    return trace, ans.text, len(think.token_ids) + len(ans.token_ids)

Forced closure is remarkably effective. A model cut off at 1,024 thinking tokens and told to answer usually produces something close to what it would have said at 3,000, because the marginal tokens late in a trace are frequently re-derivation rather than new progress.

C. Effort levels exposed to the caller. Give your API three modes — low, medium, high — that map to concrete budgets, and make product teams choose:

Mode	Think budget	Answer budget	Use for
low (or thinking off)	0	512	Classification, extraction, routing, chit-chat
medium	1,024	768	Multi-step questions, code edits, agent steps
high	4,096	1,024	Hard math/proofs, root-cause analysis, planning

For models with an explicit no-think switch (Qwen3's enable_thinking=False, or a /no_think suffix), low should use it rather than just setting the budget to zero, so the chat template stays correct.

3. Route, don't think on everything

Most production traffic does not need a reasoning model. The cheapest optimization is a router that sends the easy 70% to a plain instruct model on the same cluster.

A workable router is boring and fast:

import re

HARD_HINTS = re.compile(
    r"\b(prove|derive|why does|root cause|step[- ]by[- ]step|optimi[sz]e|plan|debug|reconcile)\b",
    re.I,
)

def route(prompt: str, tools_expected: bool) -> str:
    if len(prompt) > 4000 or tools_expected:
        return "medium"
    if HARD_HINTS.search(prompt):
        return "medium"
    return "low"

Start with rules, log the decisions, then — once you have a few thousand labelled outcomes — train a small classifier (a fine-tuned 0.5B encoder is plenty) on "did the reasoning answer beat the non-reasoning answer on this request?" Routing typically removes more cost than any kernel-level optimization you will do that quarter, because it removes the tokens entirely.

A second, complementary pattern: escalate on failure. Run low first; if a verifier (schema check, unit test, retrieval-grounding check, self-consistency disagreement) rejects the answer, retry at high. On workloads with a cheap automatic verifier this gives near-high accuracy at close to low cost.

4. Capacity planning with a long tail

Size the cluster on the token distribution, not the average. Log per-request think_tokens and answer_tokens, then:

import numpy as np

def capacity(think, answer, tpot_s=0.012, concurrency=32):
    total = np.array(think) + np.array(answer)
    for q in (50, 90, 99):
        p = np.percentile(total, q)
        print(f"p{q}: {p:6.0f} tok -> {p * tpot_s:6.1f}s per request")
    rps = concurrency / (total.mean() * tpot_s)
    print(f"sustainable ~{rps:.1f} req/s at concurrency {concurrency}")

Three knobs that matter more here than on a normal model:

• KV cache memory. Longer sequences mean fewer concurrent slots. Quantizing the KV cache to FP8 roughly doubles resident concurrency and is nearly free in quality for most reasoning workloads — measure it, but it is usually the first thing to turn on.
• Preemption. Watch your server's preemption counter. If sequences are being evicted and recomputed, you are paying for the same thinking twice; lower max_num_seqs until it stops.
• Timeouts and streaming. A user watching a spinner for 40 seconds is a failure even if the answer is right. Stream a thinking indicator, expose partial progress, and set client timeouts from p99 and not p50.

5. Handling the trace itself

Reasoning traces are a liability as much as an asset.

• Never render raw traces to end users unless you have decided to, deliberately. They contain discarded hypotheses, occasional profanity in some distills, and sometimes verbatim snippets of retrieved context you meant to keep internal.
• Strip them before they reach downstream parsers. Use the server's reasoning parser (--reasoning-parser deepseek_r1, qwen3, and friends) so the API returns reasoning_content and content as separate fields rather than one blob you regex later.
• Do not feed traces back into multi-turn history. Keep the final answers; drop the thinking. Re-feeding traces inflates prefill on every subsequent turn and, in our testing, degrades rather than improves later answers.
• Do log them, sampled and access-controlled. Traces are the single best debugging artifact you will get. Keep 1-5% with a short retention and the same PII handling as your prompts.
• Watch for unclosed tags. Truncation mid-<think> is the most common cause of "the model returned nothing" incidents. Detect it explicitly and treat it as a retry-with-closure, not a 500.

6. Prove the tokens are buying something

The point of a thinking budget is that it is tunable, which means you must measure the curve rather than pick a number. Run your eval set at several budgets and plot accuracy against mean tokens:

budgets = [0, 256, 512, 1024, 2048, 4096]
rows = []
for b in budgets:
    results = [run_eval_item(item, think_budget=b) for item in eval_set]
    acc = sum(r.correct for r in results) / len(results)
    tok = sum(r.total_tokens for r in results) / len(results)
    rows.append((b, acc, tok, acc / max(tok, 1) * 1000))

for b, acc, tok, per_k in rows:
    print(f"budget {b:5d}  acc {acc:.3f}  mean_tokens {tok:7.0f}  acc/1k tok {per_k:.3f}")

What you will almost always see: a steep rise from 0 to a few hundred tokens, a knee, then a flat or slightly declining region where extra thinking becomes overthinking — the model talks itself out of a correct first answer. Deploy at the knee, per task type, not at the model card's default.

Segment the curve by task category. In a typical client mix, extraction and classification peak at budget 0, RAG question-answering peaks around 256-512, and only genuine multi-step reasoning keeps improving past 2,000. One global budget leaves a lot of money on the table.

Also track length regressions over time: if mean thinking tokens drift up 40% after a model or prompt change while accuracy is flat, that is a cost regression and should page someone, exactly like a latency regression.

7. Stacking the usual optimizations

Reasoning workloads are decode-heavy, which changes the value of the standard toolkit:

• Speculative decoding is worth more than usual: thinking text is highly predictable, so acceptance rates on traces run high. Big latency win on low-concurrency interactive paths.
• FP8 KV cache buys concurrency, which is the binding constraint here.
• Prefix caching helps less than on RAG workloads — thinking tokens are unique per request — but still pays for long shared system prompts and multi-turn agents.
• Distillation is the endgame. Once you have logged thousands of accepted traces, distilling into a smaller student that answers directly (or thinks briefly) turns a variable 3,000-token bill into a fixed 300-token one for the subset of traffic it handles.

Checklist before you ship

• Hard total-token cap on every reasoning call, plus forced </think> closure on budget exhaustion.
• Three named effort modes wired to explicit budgets, chosen per endpoint.
• A router sending easy traffic to a non-thinking path, with logged decisions.
• FP8 KV cache enabled and preemption counters at zero under peak load.
• Reasoning parser configured server-side; traces stripped from user output and from multi-turn history.
• An accuracy-versus-budget curve per task type, re-run whenever the model or prompt changes.
• Cost dashboards on mean and p99 thinking tokens, alerting on drift.

Reasoning models are the first class of model where the prompt no longer determines the price. Treat the thinking budget as a first-class product parameter — measured, capped and routed — and you get most of the accuracy for a fraction of the tokens. Leave it unbounded and the cluster will find the tail for you.

If you are running reasoning models in production and the token bill is outrunning the accuracy gains, get in touch — budget tuning and routing are usually a short, high-return engagement.