“Three levers, and they fix different failures. Prompting fixes instruction-following — the model can do the task, it just isn’t doing it your way. Retrieval fixes missing knowledge — the model doesn’t know your data, or your data changes daily. Fine-tuning fixes consistency — the model knows how, but drifts on format or tone across thousands of calls, or you need a smaller model to hit a latency and cost target.
The order matters: prompt, then retrieval, then fine-tune. Most teams reach for fine-tuning first because it sounds like the serious answer, spend six weeks building a dataset, and find out the real problem was a vague system prompt.
In production they combine. Fine-tune for shape, retrieve for facts. What I never do is fine-tune facts into a model, because then every fact change is a retraining job.”
The deep dive — diagrams, tradeoff tables, and the follow-up trap
Match the lever to the failure
Three levers, three different failures — in the order you should try them
The symptom tells you which one you need
The mistake is almost always reaching one rung too high.
Symptom
What teams reach for
What actually fixes it
Output format wanders
fine-tune
schema-constrained decoding plus two examples
Answers are a week stale
fine-tune
retrieval with a freshness filter
Cites documents that don’t exist
fine-tune on the docs
retrieval with the id checked against the index
Tone is off-brand 1 reply in 5
more prompt rules
fine-tune on 2–5K approved replies
p95 is 4s on a frontier model
caching
distil to a small fine-tuned model
Cannot do the domain reasoning at all
fine-tune
a stronger base model — tuning does not add reasoning
What a fine-tune actually buys
The honest case for fine-tuning is rarely quality. It is holding quality while the model gets small enough to be fast and cheap.
Frontier model, prompted
8B model, fine-tuned
Format compliance
~71%
~98%
p95 latency
3.4s
0.9s
Cost / 1K replies
$4.10
$0.38
Time to change behaviour
minutes
a retrain cycle
Facts come from
retrieval
retrieval — still
REAL SYSTEM
A support-reply system had a tone problem: about one reply in five read wrong for the brand, and no amount of prompt rules held it. We fine-tuned an 8B model on ~12K human-approved replies — format compliance went ~71% → ~98%, p95 3.4s → 0.9s, and cost per thousand replies $4.10 → $0.38. Every fact in those replies still came from retrieval, so a pricing change is a reindex, not a retrain.
FOLLOW-UP TRAP
“Why not fine-tune on your docs and drop RAG entirely?” — because you retrain on every document change, you cannot cite a source, you cannot apply per-user permissions at retrieval time, and you cannot delete a fact when a customer asks you to. Fine-tuning teaches shape; retrieval supplies facts. A model with the facts baked in is confidently wrong the day the facts move.
Learn it in full:When to Fine-Tune: The Decision Framework — the prompt → retrieval → fine-tune ladder, the three questions that settle it, and why facts stay out of the weights.
