Most fine-tuning projects that fail do not fail on the training loop. The LoRA config was fine, the loss curve went down, the GPU hours were cheap — and the model came out worse than the base checkpoint on the only tasks the business cared about. Nine times out of ten the cause sits upstream: the dataset. Duplicated examples, a long tail of templated junk, two thousand rows that leaked straight out of the eval set, and a format the model was never taught to produce.
Data curation is the part of the pipeline with the worst effort-to-glamour ratio and the best return. This tutorial is the pipeline we actually run before a fine-tune: generate or collect, normalize, deduplicate, filter for quality, decontaminate against evals, balance, and freeze a versioned artifact. Everything is PyTorch-adjacent plain Python, and every stage emits numbers you can put in front of a stakeholder.
The shape of the pipeline
raw sources ──► normalize to one schema ──► exact dedup (hash)
└──► near-dup dedup (MinHash LSH) ──► quality filter (heuristics + model judge)
└──► eval decontamination ──► mixture balancing ──► versioned frozen artifact
Run the stages in that order. Dedup before quality scoring, because scoring is the expensive step and you do not want to pay it on 40% duplicates. Decontaminate last among the filters, because it is the one you must be able to re-run when a new eval set appears.
1. Normalize to one schema
Every source — support tickets, a scrape, human-labelled pairs, synthetic generations — lands in one record shape before anything else touches it:
from dataclasses import dataclass, field, asdict
@dataclass
class Example:
id: str # stable hash of the content, not a row number
messages: list # [{"role": "user"|"assistant"|"system", "content": str}]
source: str # "zendesk_2025", "synth_qwen3_v2", "human_batch_4"
meta: dict = field(default_factory=dict) # license, lang, task, difficulty
The source field is not bookkeeping; it is how you later discover that one bad generator produced 30% of your worst examples, and how you drop it without rebuilding everything. Give every example a content-derived id so that reruns are idempotent.
Tokenize with the target model's tokenizer at this stage and store the length. You need it for dedup thresholds, for filtering, and for packing later.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
def with_length(ex: Example) -> Example:
text = tok.apply_chat_template(ex.messages, tokenize=False)
ex.meta["n_tokens"] = len(tok(text).input_ids)
return ex
2. Synthetic generation that is worth keeping
When you do not have enough real data — the common case for a bespoke enterprise task — you generate it. Two rules save most projects:
Ground every generation in a real artifact. Do not ask a model for "100 customer support questions". Ask it for questions about this specific document, ticket or table, and keep the source reference in meta. Ungrounded synthetic data collapses into a handful of paraphrased templates and teaches the model a style, not a capability.
Verify, do not trust. For anything with a checkable answer — SQL, JSON extraction, code, arithmetic, a classification with known labels — run the check and discard failures. A generator at 60% validity plus a verifier beats a generator at 85% validity with no verifier, because the second one silently poisons the set.
import json, hashlib
def make_extraction_examples(doc, schema, llm, n=4):
prompt = (f"Document:\n{doc}\n\nEmit {n} JSON objects matching this schema:\n"
f"{json.dumps(schema)}\nOne per line. Use only facts in the document.")
out = []
for line in llm(prompt).splitlines():
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue # verifier stage 1: parseable
if not set(obj) <= set(schema["properties"]):
continue # verifier stage 2: schema-valid
if not all(str(v) in doc for v in obj.values() if isinstance(v, str)):
continue # verifier stage 3: grounded in source
out.append(Example(
id=hashlib.sha256((doc[:256] + line).encode()).hexdigest()[:16],
messages=[{"role": "user", "content": f"Extract fields from:\n{doc}"},
{"role": "assistant", "content": json.dumps(obj)}],
source="synth_extract_v3",
meta={"grounded_doc": doc[:64]}))
return out
Also vary the input distribution deliberately: document lengths, missing fields, ambiguous cases, and a slice of examples where the correct answer is "not present in the document". Models fine-tuned only on clean, answerable cases become confident fabricators the moment production hands them something messy.
Check the license on anything you distil from. Several popular model licenses restrict using outputs to train competing models, and that question always arrives from legal after the model is already deployed.
3. Exact and near-duplicate dedup
Exact dedup is a hash set over normalized text and takes two lines. The real work is near-duplicates: the same ticket resolved three times with a different customer name, or a generator that produced 400 variations on one sentence. Standard tool is MinHash + LSH.
import re
from datasketch import MinHash, MinHashLSH
def shingles(text, k=5):
words = re.sub(r"\s+", " ", text.lower()).split()
return {" ".join(words[i:i+k]) for i in range(max(1, len(words) - k + 1))}
def minhash(text, num_perm=128):
m = MinHash(num_perm=num_perm)
for sh in shingles(text):
m.update(sh.encode())
return m
def dedupe(examples, threshold=0.8):
lsh = MinHashLSH(threshold=threshold, num_perm=128)
kept = []
for ex in examples:
text = " ".join(m["content"] for m in ex.messages)
m = minhash(text)
if lsh.query(m): # a near-duplicate is already in the set
continue
lsh.insert(ex.id, m)
kept.append(ex)
return kept
Threshold guidance: 0.8 Jaccard over 5-word shingles is a sane default for prose. Go stricter (0.9) for code and structured outputs, where legitimate examples share a lot of boilerplate and an aggressive threshold will delete real diversity. Always log how many rows each stage removed, broken down by source — a source that loses 70% of its rows to dedup is a generator that needs fixing, not a dataset that needs filtering.
Semantic dedup is the heavier variant: embed everything, cluster with k-means, and within each cluster drop the members closest to the centroid. It catches paraphrases that MinHash misses, at the cost of an embedding pass. Worth it above roughly a million rows or when your data is heavily synthetic.
4. Quality filtering
Cheap heuristics first, because they cost nothing and remove most of the garbage:
- Length bounds on both sides: drop assistant turns under ~10 tokens or over your training
max_seq_len(they will be truncated into nonsense anyway). - Language ID, if you are training a single-language model.
- Refusal and artifact patterns:
"As an AI language model","I cannot","```"blocks that never close, repeated-character runs. - Format validity: if the task emits JSON, every assistant turn must parse.
- High n-gram repetition within a single example (a degenerate generation signature).
Then a model judge on what survives. Score 1-5 on task-specific axes — helpfulness, correctness, format adherence — and keep the top band:
RUBRIC = """Rate the assistant reply 1-5 for: (a) factual support from the user turn,
(b) completeness, (c) format adherence. Reply with only JSON: {"a":n,"b":n,"c":n}"""
def judge_keep(ex, llm, floor=4):
s = json.loads(llm(RUBRIC + "\n\n" + json.dumps(ex.messages)))
ex.meta["judge"] = s
return min(s.values()) >= floor
Two cautions. Judges are biased toward long, florid answers, so if you want terse outputs, say so in the rubric and spot-check that the judge is not quietly selecting for verbosity. And calibrate: hand-label 100 examples yourself, compare against the judge, and only then let it filter a million rows. A judge you have never audited is a random filter with good PR.
The consistent finding across the open fine-tuning literature is that a few thousand carefully filtered examples beat hundreds of thousands of unfiltered ones for instruction-style tasks. Filter hard. If you are keeping more than half of a scraped or synthetic pool, your filter is probably not doing anything.
5. Decontaminate against your evals
This is the stage teams skip and then regret publicly. If an eval question is in your training set, your eval number is fiction, and you will make a deployment decision on it.
def decontaminate(train, eval_sets, threshold=0.7):
lsh = MinHashLSH(threshold=threshold, num_perm=128)
for i, q in enumerate(eval_sets):
lsh.insert(f"eval-{i}", minhash(q))
clean, hits = [], 0
for ex in train:
user = " ".join(m["content"] for m in ex.messages if m["role"] == "user")
if lsh.query(minhash(user)):
hits += 1
continue
clean.append(ex)
print(f"decontamination removed {hits} / {len(train)} ({hits/len(train):.2%})")
return clean
Use a looser threshold here than for dedup: a false positive costs you one training row, a false negative costs you the credibility of every number you report. Re-run decontamination every time an eval set changes, and record the contamination rate in the dataset card. If it comes back at 0.00% on a web-derived corpus, your matcher is broken.
6. Balance the mixture, then freeze
A fine-tuning set is a mixture, and the mixture weights are a hyperparameter you tune like any other. Three things to control:
Task balance. If 80% of your rows are one easy task, the model will regress on the other three. Cap each task slice — downsample the dominant one rather than upsampling the rare ones, which just re-introduces duplicates.
Replay. Include 5-15% general instruction data from a broad open set. This is the cheapest known defence against catastrophic forgetting: without it, a model fine-tuned hard on your narrow task loses general chat competence, and users notice within a day.
Difficulty spread. Keep some easy examples. An all-hard set produces a model that is confidently wrong on trivial inputs.
Then freeze. The artifact is not a folder of JSONL on someone's laptop:
import json, hashlib, datetime, pathlib
def freeze(examples, path, stage_counts, mixture):
path = pathlib.Path(path); path.mkdir(parents=True, exist_ok=True)
rows = [json.dumps(asdict(e), sort_keys=True) for e in examples]
body = "\n".join(rows)
(path / "train.jsonl").write_text(body)
card = {
"n": len(examples),
"sha256": hashlib.sha256(body.encode()).hexdigest(),
"built_at": datetime.datetime.utcnow().isoformat() + "Z",
"stage_counts": stage_counts, # rows in/out of every filter
"mixture": mixture, # share per source and per task
"tokenizer": tok.name_or_path,
"token_total": sum(e.meta["n_tokens"] for e in examples),
}
(path / "card.json").write_text(json.dumps(card, indent=2))
return card["sha256"]
Log that sha256 into your training run's config alongside the model checkpoint hash. When a fine-tune behaves oddly three months later, "which exact rows did this see?" must be answerable in one lookup. Treat dataset versions the way you treat model versions: immutable, hashed, and never edited in place.
7. Feed it to the trainer
A curated set pairs with two training-side details worth getting right. Pack short examples to a fixed sequence length so you are not burning compute on padding, and mask the loss on prompt tokens so the model learns to produce answers rather than to memorize your prompts:
import torch
def collate(batch, tok, max_len=4096):
input_ids, labels = [], []
for ex in batch:
ids, lab = [], []
for m in ex.messages:
seg = tok(tok.apply_chat_template([m], tokenize=False)).input_ids
ids += seg
lab += seg if m["role"] == "assistant" else [-100] * len(seg)
input_ids.append(torch.tensor(ids[:max_len]))
labels.append(torch.tensor(lab[:max_len]))
pad = tok.pad_token_id
return {
"input_ids": torch.nn.utils.rnn.pad_sequence(input_ids, True, pad),
"labels": torch.nn.utils.rnn.pad_sequence(labels, True, -100),
}
Before you launch a multi-GPU run, decode twenty batches straight out of the collator and read them. Every dataset bug we have ever chased — missing chat template, a role swapped, the loss mask inverted, a truncation that cuts every answer in half — was visible in the first twenty decoded batches and invisible in the loss curve.
What to report
A dataset build should end with a table, not a vibe:
| Stage | Rows in | Rows out | % removed |
|---|---|---|---|
| Raw collected | — | 412,000 | — |
| Normalize + length bounds | 412,000 | 383,500 | 6.9% |
| Exact dedup | 383,500 | 291,200 | 24.1% |
| MinHash near-dedup (0.8) | 291,200 | 148,900 | 48.9% |
| Heuristic quality filter | 148,900 | 121,400 | 18.5% |
| Judge filter (min score ≥ 4) | 121,400 | 38,700 | 68.1% |
| Eval decontamination (0.7) | 38,700 | 38,310 | 1.0% |
| Mixture cap + 10% replay | 38,310 | 31,900 | — |
Those numbers are the artifact a reviewer can argue with. They also tell you where to spend next: a 49% near-dedup loss points at your generator, a 68% judge loss points at your prompt, and a contamination rate above a couple of percent means your eval numbers need a public asterisk.
Curation is unglamorous and it is where the model quality actually comes from. Build the pipeline once, make every stage emit counts, hash the output, and your next fine-tune becomes an experiment you can reason about instead of a lottery ticket.
IntelliSensei builds and audits PyTorch fine-tuning pipelines — data curation, synthetic generation with verification, decontamination and evaluation — for teams shipping custom models. If your fine-tunes are not beating the base model, the dataset is the first place we look. Get in touch.