Fine-tuning is the most expensive, slowest, and most frequently misapplied tool in the applied AI toolkit. In the last eighteen months, I have reviewed the AI roadmaps of eleven companies that had "fine-tune the model" on their near-term plan. In nine of those cases, the actual problem was retrieval — and fine-tuning would have made it worse.
This is not a knock on fine-tuning as a technique. It is a precision instrument with real uses. The problem is that teams reach for it as a first response to model underperformance, when underperformance almost always traces back to what you are giving the model, not what the model has learned.
What Fine-Tuning Actually Does (and Doesn't Do)
Fine-tuning adjusts the weights of a model on a new dataset, shaping its priors — its tendencies, its style, its response format, its domain vocabulary. It is exceptionally good at teaching a model to behave differently. It is not good at teaching a model to know something it cannot access.
If your model gives wrong answers because it lacks information, fine-tuning on more information creates an illusion of improvement — the model becomes more fluently wrong about a larger set of topics. Worse, fine-tuning on factual Q&A pairs tends to bake in the facts as of training time, which means a fine-tuned model cannot tell you when its facts have changed without another training run.
The failure mode I see most often: a team builds a domain-specific assistant. Early user feedback says the answers are inaccurate. The team's first hypothesis is "the base model doesn't know our domain." They spend six to ten weeks gathering training data, running fine-tuning, evaluating — and the accuracy improves modestly, then plateaus. Six weeks later, the product still has an accuracy problem. And now they've spent $40,000–$80,000 (engineering time, GPU compute, eval infrastructure) on the wrong fix.
The right hypothesis: the model doesn't have the right context when it generates the answer. That is a retrieval problem.
The Retrieval Failure Taxonomy
When I diagnose a RAG system that is underperforming, the failures cluster into five categories. All five are fixable without touching the model.
| Failure Type | What It Looks Like | Root Cause |
|---|---|---|
| Irrelevant chunk retrieval | Model answers confidently but incorrectly | Embedding model mismatch or poor chunking |
| Missing context | Model says "I don't have that information" for content that exists | Retrieval window too narrow, or query not matching chunk semantics |
| Context overflow | Answers get worse as documents get longer | Too many chunks retrieved, diluting the signal |
| Stale context | Model gives outdated answers | Vector index not refreshed, or no metadata filtering on recency |
| Fragmented context | Answers lack coherence across a multi-document topic | No parent-document retrieval or context window assembly |
Each of these has a specific fix. None of them require retraining the model.
Irrelevant chunk retrieval is usually a chunking problem. The naive approach — split documents into 512-token chunks on paragraph breaks — produces chunks that lose the surrounding context needed to make them meaningful. A chunk that says "This applies to the policy described above" is useless in isolation. The fix is contextual chunking: sliding windows with overlap, parent-child indexing where small chunks are retrieved but their parent document is sent to the model, or document-level summaries indexed alongside raw chunks.
Missing context is often a query-document semantic mismatch. Users ask questions in natural language; documents are written in structured, formal prose. The embedding similarity between "what happens if I cancel early" and "Section 4.2: Early Termination Penalties" may be lower than you expect, depending on your embedding model. HyDE (Hypothetical Document Embeddings) — generating a hypothetical answer and embedding that for retrieval — consistently outperforms naive query embedding for technical and formal domains.
Context overflow is a context window management problem. Retrieving the top-20 chunks and shoving them all into the prompt is how you produce confused, hedging, hallucinated answers. The model attends to whatever is in context, including the noise. A reranking step — using a cross-encoder model to score chunk-query relevance and keep only the top 3–5 — dramatically improves precision with almost no infrastructure overhead.
More context is not always better
A common instinct is to retrieve more documents to give the model "more to work with." This backfires past a threshold. Research on lost-in-the-middle effects shows that models reliably attend to the beginning and end of long contexts and often miss critical information in the middle. Five high-quality, tightly relevant chunks beat twenty loosely relevant ones almost every time.
When Fine-Tuning Is Actually the Right Answer
I do not want to overstate the case against fine-tuning. There are four scenarios where it is the correct tool.
1. Persistent stylistic or format requirements. If your application requires a specific output format — structured JSON, a particular tone, domain-specific terminology — that you cannot reliably get with prompt engineering, fine-tuning on format examples is efficient and durable. You are teaching the model how to respond, not what to know.
2. Latency and cost at extreme volume. A fine-tuned smaller model (7B–13B parameters) for a high-volume, narrow task can cost 80–90% less than using a frontier model with a long system prompt. If you have a specific, well-defined task running at millions of queries per day, fine-tuning a smaller model is economically compelling — but only after you have validated the task definition with the frontier model first.
3. True domain adaptation for specialised language. Legal, medical, and highly technical domains sometimes use terminology, abbreviations, and conventions that frontier models handle poorly even with good context. Fine-tuning on domain corpora can improve the model's baseline fluency in ways that retrieval cannot. Note: this is about language patterns, not factual knowledge.
4. Removing instruction-following noise. Some tasks are hurt by the RLHF alignment of frontier models — the tendency to hedge, to add caveats, to refuse borderline requests. For closed-domain applications where you control the input completely, fine-tuning can sharpen a model's behaviour for your specific use case.
Notice what is not on this list: "our domain knowledge is not in the model." That is a retrieval problem.
The Diagnostic I Run Before Recommending Anything
Before any conversation about fine-tuning, I run a structured failure analysis. It takes two to three days and answers these questions:
-
Where exactly are the failures? I pull 100–200 real failure cases from production logs and categorise them manually. Most teams have not done this. The category distribution tells you the root cause.
-
Is the information available in the corpus? For each failure, I check whether the correct answer exists somewhere in the source documents. If the answer is in the corpus and the model is not finding it, that is definitively a retrieval problem. If the answer is not in the corpus at all, that is a data problem — not a model problem and not a fine-tuning problem.
-
What happens with oracle retrieval? I manually assemble the perfect context for each failure case — the exact document chunks that contain the right answer — and run inference. If the model answers correctly with good context, the model is fine. If it still fails with perfect context, then we have a model problem worth investigating.
In every engagement where I have run this diagnostic, the oracle retrieval test passes for the overwhelming majority of failure cases. The model was capable. The retrieval was not.
The oracle test is the fastest way to settle the argument
If you hand the model the right context and it gives the right answer, you do not have a model problem. This test takes an afternoon to run on 50 representative cases. It resolves debates that teams have been having for weeks.
The diagnostic follows a clear decision path:
What a Properly Engineered RAG System Looks Like
The retrieval pipeline is not a single embedding lookup. It is an orchestrated sequence of steps, each of which has quality levers.
| Stage | Naive Implementation | Production Implementation |
|---|---|---|
| Chunking | Fixed-size splits, no overlap | Contextual chunking with parent-document index |
| Embedding | One generic embedding model | Domain-tuned or task-specific embedding model |
| Retrieval | Top-K by cosine similarity | Hybrid search (dense + sparse BM25) with metadata filters |
| Reranking | None | Cross-encoder reranker, top-3 to top-5 selection |
| Context assembly | Chunks concatenated as-is | Deduplication, ordering by relevance, context compression |
| Query transformation | Raw user query | HyDE, query expansion, or multi-query decomposition |
| Freshness | Static index | Incremental index updates with recency metadata |
The delta between the naive and production implementation is where most of the quality lives. I have seen teams move from 48% answer accuracy to 81% accuracy solely by improving chunking strategy and adding a cross-encoder reranker — no model change, no fine-tuning, six days of engineering work.
A production RAG pipeline is an orchestrated sequence, not a single lookup:
The Cost of Getting This Backwards
The opportunity cost of pursuing fine-tuning when you have a retrieval problem is not just the direct cost of the training run. It is the six to twelve weeks of engineering time spent preparing datasets, running experiments, building evaluation pipelines — all for a technique that addresses the wrong failure mode. Meanwhile, the product sits in a degraded state, user trust erodes, and the actual problem (retrieval) remains unsolved.
I have seen teams run three successive fine-tuning attempts, each showing diminishing returns, before finally doing the diagnostic I described above. The first diagnostic ran in two days. The retrieval fix took four more. The product problem that had persisted for five months was resolved in one week — after the team stopped working on the wrong thing.
Fine-tuning is powerful. But it is not a general-purpose fix for an AI product that is not performing. Most of the time, the model knows enough. It just is not being given what it needs to apply that knowledge to your problem.
Start with retrieval. Then, only if the oracle test fails, talk about the model.
If your AI product is underperforming and you are not sure whether you have a retrieval problem or a model problem, let's talk — book a 30-minute discovery call and I will help you run the diagnostic before you commit to an expensive training run.