Almost every other tutorial on this site ends with some version of the same instruction: evaluate honestly before you ship. This one is about the thing that instruction assumes you have. In our engagements the single most common reason an LLM feature stalls between demo and production is not the model, the GPU or the serving stack. It is that nobody can answer "is the new version better than the old one?" with anything sturdier than a few prompts typed into a chat box.
This tutorial builds an evaluation harness from scratch: a golden dataset, a runner, deterministic metrics, a calibrated judge model for the things you cannot measure with string comparison, confidence intervals so you stop chasing noise, and a CI gate that fails a pull request when quality drops. The code is plain Python and PyTorch-ecosystem friendly; it works the same whether you serve with vLLM, call a hosted API, or run a fine-tuned adapter locally.
1. Start with the golden set, not the metric
An evaluation set is a contract about what the system is for. Build it before you build the scoring.
Rules that have survived contact with real projects:
- Draw from production traffic, not your imagination. Sample real inputs (logged with consent and scrubbed of PII). Hand-written test prompts systematically miss the messy inputs that break things.
- Stratify deliberately. Bucket by the dimensions you care about: query type, document length, language, customer segment, and an explicit "known hard" bucket of past failures. Report per-bucket scores, not just an average; a mean can hide a 20-point collapse on 10% of traffic.
- 200-500 examples is usually enough to start. Below ~100 you cannot distinguish a real change from noise. Above ~1,000 the labelling cost starts to dominate before the statistics improve much.
- Freeze it and version it. The evaluation set lives in git (or a dataset registry) with a version tag. Every result you record cites the dataset version. When you add examples, bump the version and re-baseline; never quietly edit rows.
- Never train on it. Keep a separate development set for prompt iteration. The moment you tune against the golden set, it stops measuring generalization — the failure mode we describe in Why enterprises still get PyTorch wrong.
A row is just data. JSONL keeps it diffable:
{"id": "inv-0142", "bucket": "multi_page_invoice", "input": {"question": "What is the VAT total?", "doc_id": "acme-2026-03"}, "expected": {"vat_total": "1284.50"}, "notes": "figure only appears in the footer"}
2. The runner: separate generation from scoring
The most useful architectural decision in a harness is to run the system once, persist every output, and score the persisted outputs afterwards. Scoring changes far more often than generation, and re-running a 500-example generation pass every time you tweak a metric is how harnesses end up unused.
import asyncio, json, hashlib, time
from pathlib import Path
async def run_case(system, case, semaphore):
async with semaphore:
t0 = time.perf_counter()
try:
output = await system(case["input"])
error = None
except Exception as exc: # a crash is a result, not a stop
output, error = None, repr(exc)
return {
"id": case["id"],
"bucket": case.get("bucket", "default"),
"output": output,
"error": error,
"latency_s": round(time.perf_counter() - t0, 3),
}
async def generate(system, cases, run_id, concurrency=8):
sem = asyncio.Semaphore(concurrency)
results = await asyncio.gather(*(run_case(system, c, sem) for c in cases))
path = Path(f"runs/{run_id}.jsonl")
path.parent.mkdir(exist_ok=True)
with path.open("w") as fh:
for r in results:
fh.write(json.dumps(r) + "\n")
return results
Record the run's provenance alongside the outputs: model name and revision, adapter hash, prompt template hash, retrieval index version, temperature, and the dataset version. A result without provenance cannot be reproduced, and an irreproducible result is an anecdote.
Set temperature=0 (or a fixed seed) for evaluation runs unless you are specifically measuring variance. If your product runs at temperature 0.7, evaluate at 0.7 and run each case three times, then report the mean — sampling noise otherwise dwarfs the change you are trying to detect.
3. Deterministic metrics first
Use a judge model only for what cannot be checked mechanically. Far more is mechanical than teams assume:
| Task shape | Deterministic metric |
|---|---|
| Structured extraction | per-field exact match, numeric tolerance, schema validation rate |
| Classification / routing | accuracy, macro-F1, confusion matrix per bucket |
| Retrieval (RAG) | recall@k, MRR, nDCG on labelled relevant chunks |
| Code generation | unit tests pass, linter clean, compiles |
| SQL generation | result-set equality against the reference query |
| Tool / function calling | correct tool chosen, arguments schema-valid, argument exact match |
| Grounded answering | citation validity: does every cited span exist in the retrieved context? |
| Any | JSON parse rate, refusal rate, p95 latency, cost per request |
def score_extraction(case, result):
if result["error"] or not isinstance(result["output"], dict):
return {"field_accuracy": 0.0, "parsed": 0.0}
expected, got = case["expected"], result["output"]
hits = sum(1 for k, v in expected.items() if _norm(got.get(k)) == _norm(v))
return {"field_accuracy": hits / len(expected), "parsed": 1.0}
Retrieval quality deserves its own scoreboard even inside an end-to-end system: if recall@10 is 0.6, no amount of prompt engineering fixes the 40% of questions whose answer was never retrieved. The mining and evaluation workflow in Fine-tuning an embedding model for better RAG retrieval plugs directly in here.
4. LLM-as-judge, calibrated
For open-ended output — summaries, explanations, support replies — you need a model to grade. Judges are useful and also quietly biased: they prefer longer answers, prefer their own family's style, and drift when the judge model is silently upgraded. Make them trustworthy with four disciplines.
Score a rubric, not a vibe. Ask for specific binary or 1-5 criteria (factually supported by the context; answers the question asked; no invented figures; correct tone) rather than "rate this 1-10". Return structured output so it parses:
JUDGE_PROMPT = """You are grading an assistant's answer against a reference.
Question: {question}
Reference answer: {reference}
Candidate answer: {candidate}
Grade each criterion as true or false, then give one sentence of justification.
Return JSON: {{"factual": bool, "complete": bool, "no_extra_claims": bool, "why": str}}
"""
Pin and version the judge. Record the exact judge model and revision in the run provenance. When the judge changes, re-score the baseline run too, or your comparison is meaningless.
Calibrate against humans. Have a domain expert grade 50-100 examples with the same rubric, then measure agreement (Cohen's kappa) between human and judge. Above ~0.6 the judge is a useful proxy; below that, fix the rubric before trusting a single number. Re-check quarterly.
Prefer pairwise comparison for shipping decisions. Absolute scores drift; "is A better than B?" is more stable. Randomize which candidate is presented first, and run each pair in both orders to cancel position bias. Report win rate with a confidence interval.
5. Confidence intervals, or you are reading noise
On a 300-example set, 82% versus 79% is not a result. Bootstrap it:
import numpy as np
def bootstrap_ci(scores, n=10_000, alpha=0.05, seed=0):
rng = np.random.default_rng(seed)
arr = np.asarray(scores, dtype=float)
means = rng.choice(arr, size=(n, arr.size), replace=True).mean(axis=1)
return float(arr.mean()), tuple(np.quantile(means, [alpha / 2, 1 - alpha / 2]))
def paired_delta_ci(a, b, n=10_000, seed=0):
"""Same cases scored by both systems: bootstrap the per-case difference."""
rng = np.random.default_rng(seed)
d = np.asarray(a, dtype=float) - np.asarray(b, dtype=float)
idx = rng.integers(0, d.size, size=(n, d.size))
means = d[idx].mean(axis=1)
return float(d.mean()), tuple(np.quantile(means, [0.025, 0.975]))
Use the paired version whenever both systems ran the same cases — it removes case difficulty from the variance and typically halves the interval width. If the 95% interval for the delta straddles zero, you have not shown an improvement. Say so out loud; the discipline of admitting a null result is what makes the harness worth having.
6. Gate CI on it
An evaluation nobody runs decays within a month. Wire it into the pipeline:
# .github/workflows/eval.yml (fragment)
- run: python -m harness.run --dataset evalset@v7 --system candidate --out runs/pr.jsonl
- run: python -m harness.score runs/pr.jsonl --baseline runs/main.jsonl --gate gates.yaml
# gates.yaml
blocking:
field_accuracy: { min: 0.88, max_regression: 0.02 }
schema_valid: { min: 0.99 }
citation_valid: { min: 0.95 }
p95_latency_s: { max: 3.0 }
cost_per_1k_usd: { max: 4.0 }
report_only:
judge_win_rate: { min: 0.45 }
A fast subset (60-80 cases, deterministic metrics only) runs on every pull request in a couple of minutes. The full set, judge included, runs nightly on main and on release candidates. Publish the trend somewhere the whole team sees it, and keep every run artifact — six months of run files turn "it feels worse lately" into a graph.
The same harness is what you point at production traffic afterwards: sample 1% of live requests, score them with the deterministic metrics that do not need a reference, and alert on drift in schema validity, refusal rate, retrieval recall and latency.
Common failure modes
- The set that leaked. Prompts were iterated against the golden set until it saturated. Symptom: 95% offline, complaints in production. Fix: strict dev/golden split and periodic refresh from live traffic.
- The average that lies. One aggregate number over unbalanced buckets. Always report per-bucket and flag the worst bucket.
- The judge that got upgraded. A provider silently moved the judge model and every score shifted two points. Fix: pinned revisions and a baseline re-score.
- Scoring only the happy path. Timeouts, tool errors and refusals dropped from the denominator. Count every case; an exception scores zero.
- No cost or latency in the gate. A version that is one point better and three times slower is usually a regression.
Where to go next
Once the harness exists, everything else on this site becomes measurable: whether fine-tuning or RAG was the right call, whether INT8 quantization cost you accuracy, whether a migration preserved behaviour. Build it first and the rest of the roadmap stops being a matter of opinion.
We build evaluation harnesses as part of most delivery engagements, and as a standalone piece of work for teams whose LLM feature is stuck in "it demos well". If that sounds familiar, see our AI consulting and strategy and RAG systems practices, or get in touch with a description of what you are shipping.