Fine-Tuning and Serving a Tool-Calling Agent on Open Models

Every team that has shipped a RAG assistant eventually gets the same request: "can it do things, not just answer?" Book the slot, file the ticket, run the query, update the record. That is a tool-calling agent, and in 2026 it is the single most common follow-on project we get pulled into after a retrieval system goes live.

The gap between a demo agent and a production one is not prompt engineering. It is schema discipline, a fine-tuned model that emits valid calls under your own tool names, a serving stack that enforces grammar rather than hoping, and an evaluation harness that scores trajectories rather than strings. This tutorial walks the whole path on open models with PyTorch.

When you need a fine-tuned tool-caller at all

Frontier hosted models are good at function calling out of the box. Fine-tune your own when at least two of these are true:

• You have many tools (15+) or deeply nested argument schemas. Prompt-stuffed tool lists eat context and accuracy degrades fast past a dozen or so.
• Latency or cost per call matters. An agent turn is not one generation; a five-step task is five or more. A 7B model you serve yourself at 40ms per step beats a hosted call at 800ms.
• Your arguments are domain-specific. Internal SKU formats, ICD codes, tenant identifiers. A general model guesses; a tuned one learns the shapes.
• Data cannot leave your estate. The usual regulated-industry constraint.

If none of those hold, use a hosted model and spend your effort on the tool layer. That is genuine advice, and we give it often.

Step 1: design the tool schema before you touch a GPU

Nearly every failed agent project we are called into has the same root cause: the tools are bad, and the model is being blamed. Rules we apply:

One tool, one verb, one outcome. update_order(order_id, action, payload) where action switches between six behaviours is not one tool, it is six tools wearing a trench coat. Split it. Models select far better among many narrow tools than among few overloaded ones.

Make illegal arguments unrepresentable. Use enums, not free strings. Use ISO dates, not "next Tuesday". Every constraint you put in the JSON Schema is a constraint the decoder can enforce for you later.

Return structured errors the model can act on. A tool that returns {"error": "not_found", "hint": "no customer with that id; try search_customers"} recovers. One that returns a 500 stack trace loops.

Keep the result payload small. Tool results are appended to context every turn. Return the ten fields the agent needs, not the row.

SEARCH_ORDERS = {
    "type": "function",
    "function": {
        "name": "search_orders",
        "description": "Find orders for a customer within a date range. Returns at most 20 summaries.",
        "parameters": {
            "type": "object",
            "properties": {
                "customer_id": {"type": "string", "pattern": "^CUS-[0-9]{8}$"},
                "status": {"type": "string",
                           "enum": ["open", "shipped", "delivered", "cancelled"]},
                "since": {"type": "string", "format": "date"},
            },
            "required": ["customer_id"],
            "additionalProperties": False,
        },
    },
}

additionalProperties: False and tight pattern/enum fields are not decoration. They become the grammar that makes malformed calls impossible at serving time.

Step 2: build the training data from traces, not imagination

Hand-written examples produce agents that work on hand-written tests. What you want is multi-turn trajectories: user turn, assistant tool call, tool result, assistant tool call, ..., final answer.

Three sources, in order of value:

1. Replayed production logs. If a hosted model is already driving your tools, log every trajectory with the tool result and whether the task succeeded. This is the highest-quality corpus you will ever have, and it costs nothing but instrumentation.
2. Synthetic generation against a real sandbox. Have a strong model attempt generated task descriptions against a live sandbox of your tools. Keep only trajectories where the end state is verifiably correct. Verification against real state is what separates this from hallucinated data.
3. Public function-calling sets (xLAM, ToolACE, Glaive and friends) mixed in at 10–20% to preserve general tool ability and stop catastrophic forgetting of formats you did not cover.

Store trajectories in the chat format your tokenizer's template already understands, so the tool-call block is rendered by apply_chat_template rather than by string concatenation you invented:

messages = [
    {"role": "system", "content": "You are an order support agent."},
    {"role": "user", "content": "Has CUS-00481920 had anything shipped since March?"},
    {"role": "assistant", "tool_calls": [{
        "type": "function",
        "function": {"name": "search_orders",
                     "arguments": '{"customer_id":"CUS-00481920",'
                                  '"status":"shipped","since":"2026-03-01"}'}}]},
    {"role": "tool", "name": "search_orders",
     "content": '{"orders":[{"id":"ORD-771","shipped_on":"2026-03-14"}]}'},
    {"role": "assistant", "content": "Yes — order ORD-771 shipped on 14 March."},
]
text = tok.apply_chat_template(messages, tools=[SEARCH_ORDERS], tokenize=False)

Two data details that decide whether this works:

• Include negative and refusal trajectories. Tasks with no applicable tool, ambiguous requests that should trigger a clarifying question, tools that return errors and must be retried differently. An agent trained only on happy paths will invent a tool call for every input, because it has never seen an example where not calling one was correct.
• Balance step counts. If 90% of your training trajectories are single-call, the model will try to finish every task in one call.

Step 3: fine-tune with loss on assistant turns only

The mechanics are the LoRA/QLoRA recipe you already know — the agent-specific part is masking. Tool results are inputs, not targets. Train on tool-result tokens and the model learns to hallucinate plausible API responses instead of waiting for real ones. This is the single most common bug in home-grown agent tuning.

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model=model,                       # 7B-ish base, LoRA r=32 on attn + MLP
    train_dataset=ds,                  # rendered chat text
    args=SFTConfig(
        max_length=8192,               # trajectories are long; do not truncate mid-call
        packing=False,                 # packing across trajectories corrupts turn boundaries
        completion_only_loss=True,
        bf16=True,
        gradient_checkpointing=True,
        num_train_epochs=2,
        learning_rate=1e-4,
    ),
)
trainer.train()

Sanity checks before you spend the GPU hours: render three training samples back out and read them; assert the label mask is -100 everywhere except assistant spans; confirm no sample is silently truncated (a truncated trajectory that cuts a JSON argument block in half teaches the model to emit invalid JSON).

Two epochs is usually right. Tool-calling tunes overfit quickly — by epoch four the model tends to memorize argument values from training tasks.

Step 4: serve with constrained decoding, not hope

A tuned model emits valid JSON almost always. In an agent, "almost" compounds: 99% per call is 95% over a five-step task. Fix it at the decoder. vLLM and SGLang both support guided decoding driven by your JSON Schema, which makes invalid output structurally unreachable.

vllm serve ./merged-agent-7b \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --max-model-len 16384

resp = client.chat.completions.create(
    model="merged-agent-7b",
    messages=messages,
    tools=[SEARCH_ORDERS, GET_CUSTOMER, CREATE_TICKET],
    tool_choice="auto",
    temperature=0.0,
)

Notes from production:

• Match the parser to the template you trained on. If you fine-tuned on Hermes-style <tool_call> blocks, serve with the Hermes parser. Mismatched parser and template is the number-one cause of "the model works in a notebook but the server returns empty tool calls".
• Greedy decoding for tool selection. Sampling buys you nothing when the target is a structured call; it buys you wrong tool names.
• Prefix caching earns its keep here. The system prompt plus tool schemas are identical on every turn of every conversation; with prefix caching enabled, step two onwards reuses that KV cache.
• Schema validity is not correctness. Constrained decoding guarantees the call parses. It does not guarantee the right tool with the right arguments. That is what evaluation is for.

Step 5: the runtime loop, with the guardrails that matter

def run_agent(user_msg, tools, registry, max_steps=8, budget_tokens=30_000):
    messages = [SYSTEM, {"role": "user", "content": user_msg}]
    used = 0
    for step in range(max_steps):
        out = client.chat.completions.create(
            model=MODEL, messages=messages, tools=tools,
            tool_choice="auto", temperature=0.0)
        msg = out.choices[0].message
        used += out.usage.total_tokens
        messages.append(msg)
        if not msg.tool_calls:
            return msg.content, messages
        for call in msg.tool_calls:
            result = registry.invoke(call, timeout=10)   # validates + executes
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "name": call.function.name,
                             "content": truncate(result, 2000)})
        if used > budget_tokens:
            return escalate("token budget exceeded"), messages
    return escalate("step limit reached"), messages

Non-negotiables in that loop:

• A step cap and a token budget. Every agent loop eventually hits a state where it calls the same tool forever. Cap both, and make exceeding the cap an escalation event you can alert on, not a silent empty answer.
• Idempotency keys on every write tool. Retries happen — at the HTTP layer, in your own error handling, in the model's head. create_ticket called twice with the same key must create one ticket.
• A human gate on irreversible actions. Refunds, deletions, outbound email. The agent proposes; a person or a policy engine approves. We have never regretted this and have several times been called in after a team skipped it.
• Truncate tool results. Two thousand characters, then a pointer. Unbounded results blow the context window on step three.
• Log the whole trajectory. Every message, every tool result, every latency. This is your next training set (step 2) and your only debugging surface.

Step 6: evaluate trajectories, not strings

Text similarity metrics are meaningless here. Score four things, on a frozen suite of 150–300 tasks with verifiable end states:

Metric	Definition	Why it matters
Task success	End state in the sandbox matches expected	The only number the business cares about
Tool-selection accuracy	Correct tool chosen at each step	Localizes failures to selection vs. arguments
Argument exact/semantic match	Args correct given the schema	Catches enum and date-format regressions
Steps to success	Median call count vs. an oracle	Detects flailing that still eventually succeeds

Add two adversarial slices: no-op tasks where the correct behaviour is to answer directly or ask a question (measures over-calling), and broken-tool tasks where a tool returns an error (measures recovery). Both catch regressions that the happy-path suite never will.

Run the suite against a sandbox with resettable state, in CI, on every model or prompt change. If you have already built the LLM evaluation harness pattern, this is the same machinery with an environment attached.

Where reinforcement learning fits

Once the SFT model is competent and you have a sandbox that can verify end states, tool-use tasks become a near-perfect fit for verifiable-reward post-training: reward 1 if the end state is correct, 0 otherwise, with small penalties for extra steps and invalid calls. That is exactly the setting GRPO was built for, and it is where we see the biggest remaining gains on multi-step tasks — but only after SFT. RL on a model that cannot yet emit valid calls just burns GPU hours.

Failure modes we see most often

• Tool sprawl. Forty tools registered, six ever used. Route: split the agent by domain, or add a retrieval step that selects the ten relevant schemas per turn.
• Training on tool-result tokens. Model hallucinates API responses and never waits for the real ones. Check your loss mask.
• Parser/template mismatch at serving time. Silent empty tool calls.
• No no-op examples. Model calls a tool for "thanks, that's all".
• Evaluating on strings. Green dashboards, unhappy users.
• Unbounded loops with write access. The expensive one. Step caps and idempotency keys are cheaper than the incident.

The realistic budget

For a 15-tool domain agent on a 7B base: two to three weeks to instrument logging and build the sandbox, one week for data curation, a day or two of LoRA fine-tuning and iteration on a single H100, and one to two weeks for the eval harness and runtime hardening. The fine-tune is the small part. It nearly always is.

If you are standing up a tool-calling agent on your own models — or rescuing one that loops, over-calls or quietly emits malformed JSON in production — get in touch. We do this work end to end, from tool schema review through fine-tuning, serving and the evaluation harness that keeps it honest.