"Should we fine-tune a model or build RAG?" is the most common question on our discovery calls, and it is usually the wrong question, because the two do different jobs. This post gives a framework for deciding, a comparison across the four dimensions that matter, and three anonymized scenarios from recent engagements where the answer was fine-tune, RAG, and both.
What each one actually changes
Retrieval-Augmented Generation supplies the model with knowledge at request time: relevant passages from your documents, placed in the prompt. The model itself does not change. It is the right tool when the answer depends on facts the model was not trained on, facts that change, or facts that differ per user.
Fine-tuning changes the model's behaviour: the format it produces, the style it writes in, the tasks it is good at, the domain vocabulary it handles fluently. It is a poor way to inject facts (a model fine-tuned on your documents still hallucinates about them) and an excellent way to make a model reliably do a specific job.
The short rule: knowledge → RAG; behaviour → fine-tune; most production systems → RAG first, a light fine-tune second.
The decision matrix
| Dimension | RAG | Fine-tuning | Both |
|---|---|---|---|
| Cost to build | Low to medium: ingestion, indexing, evaluation set | Medium: data preparation, training runs, evaluation | Medium to high |
| Cost to run | Higher per request: retrieval plus a longer prompt | Lower per request: shorter prompts, smaller model often suffices | Medium |
| Latency | Adds a retrieval step (tens of ms) and longer prompts | Lowest; a small fine-tuned model can be very fast | Medium |
| Accuracy on facts | High when retrieval is good; cites sources | Low; memorization is unreliable | High |
| Accuracy on format/task | Depends on prompting; brittle | High and consistent | High |
| Freshness | Immediate: update the index | Requires retraining | Immediate for facts |
| Governance | Strong: permissions at retrieval, citations, auditable | Weaker: knowledge is baked in, hard to delete | Strong |
| Data needed | Documents plus a few hundred labelled queries | Thousands of examples of the desired behaviour | Both |
Scenario A: a support assistant over a product knowledge base
A software company wanted an assistant that answers customer questions from 4,000 help-centre articles and release notes, updated weekly.
The knowledge changes weekly and must be cited; hallucinated product behaviour is a liability. That is RAG. We built hybrid retrieval with a reranker, fine-tuned the embedding model on historical search logs (recall@5 rose substantially), and enforced article-level permissions at retrieval time. The generation model was an off-the-shelf open-weight instruction model with a carefully engineered prompt. No fine-tuning of the LLM was needed, and the team can update the index without a model release.
Answer: RAG.
Scenario B: structured extraction from clinical documents
A healthcare analytics firm needed to extract a fixed schema of fields from scanned intake forms and referral letters, at high volume, with strict format compliance and no external API calls.
The knowledge is in the document itself; what the model must learn is the schema, the domain vocabulary and the exact output format, and it must run on-premises cheaply. That is fine-tuning. We fine-tuned an 8B open-weight model with QLoRA on 6,000 labelled documents, added constrained decoding so the output is always valid JSON, and served it quantized on a single GPU. Extraction accuracy exceeded the prompted frontier-model baseline at a fraction of the per-document cost, and no data left the building.
Answer: fine-tune.
Scenario C: an internal engineering assistant
An enterprise wanted an assistant for its engineers that answers questions about internal systems from design documents, runbooks and code, and also follows house conventions when it drafts runbook steps or incident summaries.
Two requirements, two tools. The factual side (what does service X do, where is the runbook) is RAG over the document corpus with access control. The behavioural side (write incident summaries in our template, use our terminology, refuse to speculate about production state) came from a modest instruction fine-tune plus a DPO preference pass on a few hundred ranked examples. Either alone fell short: RAG-only drafts ignored the house format; fine-tune-only answers were confidently out of date.
Answer: both, RAG first.
How to decide for your case
- Write down five representative requests. For each, ask: does a correct answer require information the model could not have seen? If yes for most, you need retrieval.
- Ask whether a correct answer also requires a specific format, tone or task skill the model gets wrong when prompted. If yes, you need a fine-tune, and it is usually a small one.
- Check governance: if content must be deletable, permissioned or cited, retrieval is mandatory regardless of the above.
- Estimate run cost at your volume. Long RAG prompts on a large model add up; a fine-tuned small model may be an order of magnitude cheaper per request.
- Prototype the retrieval half first. It is cheaper, it reveals how good your documents are, and it produces the evaluation set you will need either way.
We build both: see RAG systems and LLM fine-tuning and customization. If you would like help working through the framework for your use case, contact us and bring those five requests.