1:1 mentoring with Big Tech AI engineers
Q5

When do you fine-tune instead of adding retrieval or changing the prompt?

Fundamentals & Architecture

Fine-TuningRAGPrompt EngineeringArchitecture

Asked at OpenAI · Databricks · Scale AI

How to Answer

“Three levers, and they fix different failures. Prompting fixes instruction-following — the model can do the task, it just isn’t doing it your way. Retrieval fixes missing knowledge — the model doesn’t know your data, or your data changes daily. Fine-tuning fixes consistency — the model knows how, but drifts on format or tone across thousands of calls, or you need a smaller model to hit a latency and cost target.

The order matters: prompt, then retrieval, then fine-tune. Most teams reach for fine-tuning first because it sounds like the serious answer, spend six weeks building a dataset, and find out the real problem was a vague system prompt.

In production they combine. Fine-tune for shape, retrieve for facts. What I never do is fine-tune facts into a model, because then every fact change is a retraining job.”

The deep dive — diagrams, tradeoff tables, and the follow-up trap

Match the lever to the failure

Three levers, three different failures — in the order you should try them
Try them in this order 1 · Prompting fixes: not following instructions fixes: wrong shape on a good day cost: an afternoon reversible: instantly 2 · Retrieval fixes: doesn’t know your data fixes: facts change daily cost: days, plus an index to run reversible: reindex 3 · Fine-tuning fixes: tone/format drift at scale fixes: frontier latency and price cost: weeks, 2–10K labelled rows reversible: retrain None of the three add reasoning the base model does not have If the model cannot do the task once, by hand, with the answer sitting in front of it — that is a model choice, not a tuning problem. Most teams skip to 3 because it sounds serious, then find the bug was in 1.

The symptom tells you which one you need

The mistake is almost always reaching one rung too high.

SymptomWhat teams reach forWhat actually fixes it
Output format wandersfine-tuneschema-constrained decoding plus two examples
Answers are a week stalefine-tuneretrieval with a freshness filter
Cites documents that don’t existfine-tune on the docsretrieval with the id checked against the index
Tone is off-brand 1 reply in 5more prompt rulesfine-tune on 2–5K approved replies
p95 is 4s on a frontier modelcachingdistil to a small fine-tuned model
Cannot do the domain reasoning at allfine-tunea stronger base model — tuning does not add reasoning

What a fine-tune actually buys

The honest case for fine-tuning is rarely quality. It is holding quality while the model gets small enough to be fast and cheap.

Frontier model, prompted8B model, fine-tuned
Format compliance~71%~98%
p95 latency3.4s0.9s
Cost / 1K replies$4.10$0.38
Time to change behaviourminutesa retrain cycle
Facts come fromretrievalretrieval — still
REAL SYSTEM

A support-reply system had a tone problem: about one reply in five read wrong for the brand, and no amount of prompt rules held it. We fine-tuned an 8B model on ~12K human-approved replies — format compliance went ~71% → ~98%, p95 3.4s → 0.9s, and cost per thousand replies $4.10 → $0.38. Every fact in those replies still came from retrieval, so a pricing change is a reindex, not a retrain.

FOLLOW-UP TRAP

“Why not fine-tune on your docs and drop RAG entirely?” — because you retrain on every document change, you cannot cite a source, you cannot apply per-user permissions at retrieval time, and you cannot delete a fact when a customer asks you to. Fine-tuning teaches shape; retrieval supplies facts. A model with the facts baked in is confidently wrong the day the facts move.