One changes what the model knows. The other changes how it behaves. Almost every expensive mistake in this decision comes from confusing the two.
The question arrives in the same shape every time: “should we do RAG or fine-tune?” It is asked as if the two were competing implementations of one idea, the way Postgres and MySQL are. They are not. They fix different defects, on different timescales, with different failure modes — and picking the wrong one is not a small mistake. A fine-tune aimed at a retrieval problem costs weeks of labelling and produces a model that is fluent, confident and permanently out of date.
The right first question takes ten seconds to answer: is the model missing a fact, or missing a habit?
Retrieval changes what is in the context window. Fine-tuning changes what the model does with whatever is in the window. That is the whole distinction, and it survives every framework, provider and model generation you will use it under.
The split that decides it
Write down the failure you actually observed — a real answer, from a real query, that was wrong. Then put it on one side of this line.
What each one actually changes
Retrieval — knowledge, assembled per query
You keep the documents outside the model, find the relevant few thousand tokens at question time, and paste them into the window before the model answers. The model’s weights never move. Everything you know about context engineering applies, because retrieval is just the most automated way of deciding what goes in the window.
Fine-tuning — behavior, baked into the weights
You show the model a few thousand examples of the input–output pair you want and nudge its parameters until that pair is what it produces by default. It does not add a knowledge store; it moves a probability distribution. Whatever facts leak in are frozen at the moment training stopped, with no way to cite, filter or expire them.
The third option, which wins more often than either
Before both: a better prompt, a few examples in it, a schema constraint on the output. Not glamorous, and roughly free. A surprising share of “we need to fine-tune” is a system prompt that never said what good looked like, and a surprising share of “the format keeps drifting” is structured output that was never turned on.
Side by side, on the dimensions that decide
| RETRIEVAL (RAG) | FINE-TUNING | |
|---|---|---|
| Fixes | Missing, private or changing facts | Wrong shape, tone, or judgement |
| Time to first result | Hours to days | Weeks — most of it labelling |
| Cost to update one fact | Re-index one document, seconds | A new training run over the whole set |
| Freshness | As fresh as the index | Frozen at the end of training |
| Provenance | Native — you have the chunk you sent | None — the model cannot show its source |
| Per-query token cost | Higher — retrieved context on every call | Lower: shorter prompts, fewer examples |
| Added latency | A retrieval hop, plus reranking if used | None — often faster, on fewer input tokens |
| Data you need | The documents you already own | Hundreds to tens of thousands of labelled examples |
| Per-tenant access control | A filter on the query — easy | Impossible — a weight cannot be unlearned per user |
| Who can debug it | Anyone — read the retrieved chunks | Whoever owns the training pipeline |
| Signature failure | Retrieved the wrong chunk, answered honestly | Answered fluently, from a world that moved on |
The access-control row is the one that ends arguments in regulated shops. If two customers must never see each other’s data, their data cannot be in the same set of weights — and one fine-tune per tenant is not a plan. That constraint sits upstream of every performance argument. See multi-tenant isolation.
Paste the missing paragraph into the prompt by hand and re-ask. If the answer becomes correct, you have a retrieval problem, and no amount of fine-tuning will keep that paragraph current. If the answer is still shaped wrong with the fact sitting right there in the window, now you have a behavior problem.
Run the ladder before you run the training job
Each rung costs roughly an order of magnitude more than the one below it, in engineering time and in the number of people who have to stay involved. Climb one rung only when you have measured the one below it and watched it fail.
Four scenarios, worked
1 · “Answer questions about our internal docs”
Retrieval, and it is not close. The content changes weekly, answers need a citation to be trusted, and different teams may see different documents. All three of those are things weights cannot do. The work is not choosing RAG — that took ten seconds — it is chunking, retrieval and reranking, and an eval set built from real questions rather than from the documents themselves. Which library does that work for you is a separate decision, and a smaller one than it looks: see LangChain vs LlamaIndex vs LangGraph.
The fine-tuning version of this project ships a model that answers December’s policy in March, with total confidence and no way to tell.
2 · “It won’t stay in our output format”
Neither, then fine-tuning. Turn on constrained output first — if the shape is a schema, enforce the schema instead of asking for it. Fine-tuning earns its keep when the target is a style a schema cannot express: the house voice of a support reply, a radiology report’s hedging conventions, a five-line commit message your team recognizes. That is a behavior, it is stable across quarters, and it is expensive to specify in words — the exact profile fine-tuning is good at.
3 · “It doesn’t understand our jargon”
Usually retrieval, at the embedding layer. “Doesn’t understand our jargon” nearly always decomposes into retrieval misses the right chunk because our vocabulary is not the general one. A glossary in the index, hybrid keyword+vector search, and a domain-adapted embedding model fix more of this than a fine-tuned generator will — and you can tell them apart in an hour by checking whether the right chunk was retrieved at all. See embeddings and indexing.
4 · “It works, but it’s too slow and too expensive at volume”
Fine-tuning — the one case where it is clearly the right tool. Take the big model’s outputs on your real traffic, train a small model on them, and serve the small one. You trade generality you were not using for latency and cost you were paying for. This is distillation, and it is the highest-return fine-tune most teams ever run — because the labels already exist, produced by the system you are replacing. Before you commit, check that caching and prompt caching have not already taken the same win for a week of work.
The cost shape, not the cost
Provider prices move too fast to be worth printing. The shape does not: retrieval is cheap to build and pays rent forever, fine-tuning is expensive to build and then cheap to run.
| Cost line | RETRIEVAL | FINE-TUNING |
|---|---|---|
| Up front, engineering | Ingestion, chunking, index, eval | Label pipeline, training runs, eval, serving |
| Up front, human | Near zero — the docs exist | The dominant line — someone writes the examples |
| Per query | 1–3K more input tokens, on every single call | 500–2K fewer tokens — the instructions moved into the weights |
| When content changes | Re-embed the changed documents | Nothing works until the next training run |
| When the model improves | Swap the model, keep the index | Retrain, re-evaluate, re-deploy |
| Ongoing owner | Whoever owns the data pipeline | Whoever owns the ML pipeline — often nobody, by month six |
That last row is the honest one. Retrieval degrades visibly — someone notices a stale answer and re-indexes. A fine-tune degrades invisibly, and the person who ran it has changed teams. Put the recurring cost in the decision, not just the build cost: see cost, latency and quality tradeoffs and when to fine-tune.
How each one fails, and what the symptom means
| Symptom | Usual cause | Where to look |
|---|---|---|
| The right document exists, the answer is wrong | Retrieval returned the wrong chunk — it reads as a generation problem and is not one | Chunking, reranking |
| Confidently stale answers | A fact that lives in weights instead of an index | Move the fact to retrieval |
| Format drifts on long outputs | The format is an instruction, not a constraint | Structured output |
| Great in eval, poor in production | The eval set was written from the documents, not from real queries | RAG evaluation, RAGAS |
| Fine-tune got better at one thing, worse at everything | Too narrow a training set, or too aggressive a run | LoRA / QLoRA, data preparation |
| Quality fine, latency awful | Retrieval and reranking on every call, including repeats | Semantic caching |
The artifact that ends the argument
This decision gets made in meetings far more often than it gets measured, and it is not an expensive thing to measure. One eval set drawn from real user queries, four arms, one afternoon.
# 150 questions written from real logs -- NOT written from the docs,
# which is the mistake that makes every RAG eval look great.
questions = load("eval/real_queries.jsonl")
ARMS = {
"baseline": lambda q: model(SYSTEM, q),
"prompt+": lambda q: model(SYSTEM_WITH_EXAMPLES, q),
"rag": lambda q: model(SYSTEM, q, context=retrieve(q, k=6)),
"finetuned": lambda q: tuned_model(SYSTEM_SHORT, q),
}
def score(arm) -> float:
return sum(judge(arm(q.text), q.expected) for q in questions) / len(questions)
for name, arm in ARMS.items():
print(f"{name:>10} {score(arm):.0%}")
# baseline 41%
# prompt+ 58% <- free, and it closed 40% of the gap
# rag 79%
# finetuned 53% <- fluent, wrong: it never saw last quarter's policy
Two numbers in that output decide the project. prompt+ tells you how much of the problem was never a model problem. The gap between rag and finetuned tells you which kind of gap you actually had — and if finetuned loses to rag on a knowledge benchmark, that is not a bad fine-tune, that is the wrong technique applied competently.
The combination that usually wins
Once a system is past its first year, the answer is frequently “both” — but with a strict division of labor, and in a specific order.
Both, done badly
Fine-tune on the documents to “teach it our domain”, then add retrieval later when the answers go stale. Now the weights and the index disagree, and when they do, the model prefers itself. Every content update needs a training run to stay consistent, so it does not get one. Nobody can tell whether a wrong answer came from a bad chunk or a baked-in memory.
Both, done well
Retrieval owns every fact. The fine-tune sees zero domain facts and only teaches the habit: the output shape, the refusal boundary, the house voice, how to say “the documents do not cover this”. Content changes need a re-index, not a retrain. When an answer is wrong you look at the chunks first, and they are usually the answer.
Said as a rule: never fine-tune on anything that has an expiry date. If a fact could be revised by someone in another department next quarter, it belongs in an index.
Decide in five minutes
Five questions, in order. The first “yes” is your answer.
- Does the content change on a calendar? → Retrieval. Nothing else survives the change.
- Does an answer need a citation, or per-user filtering? → Retrieval. Weights cannot cite or filter.
- Would pasting the right paragraph into the prompt fix it? → Retrieval, and you just proved it in one query.
- Is it a stable habit that is hard to write down in words? → Fine-tuning, once few-shot has failed on a real eval set.
- Is the quality already right and only cost and latency wrong? → Fine-tune a small model on your own traffic — after you have checked caching.
If none of the five is a clear yes, you have not localized the failure yet, and any technique you pick will be a guess wearing a budget.
RAG and fine-tuning are not rivals, and the teams that treat them as rivals lose a quarter to the answer. Facts go in an index because facts expire; habits go in the weights because habits do not. Almost every project that regretted its choice made it before it had an eval set — and almost every project that was happy with it built the eval set first, then let the numbers pick.
The techniques will keep changing shape — adapters get cheaper, context windows get longer, retrieval gets smarter. The split does not. Ask what expired, and you will pick right under whatever the tooling looks like next year.