The question comes up in almost every enterprise AI project I work on: "Should we fine-tune the model or use RAG?" It's framed as a binary choice, but it's actually a more nuanced decision that depends on what problem you're trying to solve, what data you have, and what resources you're willing to invest.
Understanding when to use each approach — and when to combine them — is one of the most important architectural decisions in enterprise LLM deployment.
The Core Distinction
Retrieval-Augmented Generation (RAG) doesn't change the model. Instead, it augments the model's input: at inference time, relevant documents are retrieved from an external knowledge base and added to the prompt context. The model reasons over the retrieved information plus the user's query to generate a response.
Fine-tuning changes the model itself. You continue training a base model on your domain-specific data, adjusting the model's weights so it internalises your knowledge, style, or domain-specific behaviour.
- →No changes to model weights
- →Knowledge is in the retrieval store — update without retraining
- →Transparent: you can see exactly what was retrieved
- →Works well for large, changing knowledge bases
- →Grounding reduces hallucination on factual questions
- →Context window limits how much can be retrieved
- →Retrieval quality directly determines answer quality
- →Lower cost: no GPU compute for training
- →Modifies model weights with domain data
- →Internalises knowledge — no retrieval step
- →Better for style, tone, and format adaptation
- →Can teach the model new tasks and reasoning patterns
- →Cannot update knowledge without retraining
- →Risk of catastrophic forgetting of general capability
- →Requires high-quality training data
- →Significant compute cost (GPU hours)
When RAG Is the Right Choice
RAG is the right default for most enterprise knowledge base and document Q&A use cases. Choose RAG when:
1. Your knowledge is large, dynamic, or proprietary
If you want the model to answer questions about your product documentation, internal policies, customer contracts, or regulatory filings — and that content changes regularly — RAG is the appropriate architecture.
Fine-tuning would require retraining the model every time a policy changes. RAG requires updating the vector database (a much simpler operation).
2. You need citations and traceability
RAG can return the source documents that grounded a response. This is critical for enterprise use cases where users need to verify answers (legal, compliance, medical) or where audit trails are required.
Fine-tuned models cannot tell you which training example produced a particular output.
3. You're building on a frontier model (GPT-4, Claude, Gemini)
Leading proprietary models are not available for fine-tuning (or fine-tuning has limited capability). RAG is the only option for customising behaviour without switching to an open model.
4. You need to reduce hallucination on factual questions
RAG grounding dramatically reduces hallucination on factual questions by providing the model with authoritative reference material. Without grounding, even the best models confabulate details.
The Architecture of a Production RAG System
A production RAG pipeline has more complexity than "embed documents, retrieve, generate":
Source documents are ingested, cleaned, and split into chunks. Chunking strategy significantly affects retrieval quality — chunks that are too small lose context; too large dilute the signal.
- →Semantic chunking (split at sentence/paragraph boundaries)
- →Overlapping chunks for boundary spanning
- →Metadata extraction (title, date, section, source)
Each chunk is converted to a vector embedding using an embedding model. Embeddings are stored in a vector database for approximate nearest-neighbour search.
- →text-embedding-3-large (OpenAI) or Azure OpenAI
- →Hybrid indexing: vector + keyword (BM25) for best recall
- →Azure AI Search natively supports hybrid retrieval
User queries may need transformation before retrieval. Query expansion, hypothetical document embedding (HyDE), and query decomposition all improve retrieval quality.
- →Query rewriting for better semantic match
- →HyDE: generate a hypothetical answer to retrieve against
- →Sub-query decomposition for multi-part questions
Retrieve the top-K candidates, then re-rank using a cross-encoder model for higher precision. Filtering by metadata narrows the search space.
- →Initial retrieval: top 20 to 50 candidates
- →Re-ranking: cross-encoder scores all candidates against query
- →Final selection: top 3 to 5 chunks for context
Retrieved context is injected into the prompt alongside the user query. The generation prompt instructs the model to answer based on the provided context and cite sources.
- →System prompt: answer based only on the provided context
- →Include source metadata with each chunk
- →Return citations alongside the answer
When Fine-Tuning Is the Right Choice
Fine-tuning addresses different problems than RAG. Choose it when:
1. You need to change the model's behaviour, not just its knowledge
If you need the model to adopt a specific format (always return JSON with a defined schema), follow a specific reasoning style, or behave consistently in a way that's difficult to specify in a prompt — fine-tuning is the right tool.
Example: A customer support model that must always respond in a specific tone, extract structured data from unstructured input, or follow a multi-step reasoning pattern. Prompting can get you close; fine-tuning makes it reliable.
2. You have a specialised domain with unusual vocabulary or concepts
A model fine-tuned on clinical notes understands medical terminology and clinical reasoning patterns that a general model handles poorly. A model fine-tuned on legal contracts understands boilerplate, exceptions, and standard clause structures.
This is different from having lots of documents — RAG can handle lots of documents. It's about whether the model needs to understand the domain, not just retrieve from it.
3. You need to reduce prompt length and inference cost at scale
If your application makes millions of API calls, a fine-tuned smaller model may deliver equivalent quality at a fraction of the cost of prompting a large frontier model. Fine-tuning 7B or 13B parameter models for specific tasks often achieves GPT-4-level performance on those specific tasks.
4. Latency requirements exceed frontier model capabilities
Fine-tuned smaller models can be deployed locally or on dedicated GPU infrastructure with latency characteristics that hosted frontier models cannot match.
Fine-tuning requires excellent training data
The most common fine-tuning failure: poor-quality training data. Fine-tuning amplifies the patterns in your training data — if your data has errors, biases, or inconsistencies, the fine-tuned model will too. Invest heavily in data quality and curation before training. 1,000 high-quality examples outperforms 100,000 mediocre ones.
The Combined Architecture: RAG + Fine-Tuned Model
For the highest performance on domain-specific tasks, combining RAG with a fine-tuned model is often the best approach:
- Fine-tune for behaviour: Train the model on your domain's reasoning patterns, output format, and task-specific examples
- RAG for knowledge: Keep factual knowledge in the retrieval store so it can be updated without retraining
This gives you the best of both worlds: a model that behaves correctly for your domain, grounded in current knowledge that can be updated without retraining.
Example: A financial analysis assistant might use:
- A fine-tuned model trained on financial reasoning patterns and report structures
- RAG over a continuously updated store of financial reports, earnings calls, and market data
The Decision Framework
| Situation | Recommendation |
|---|---|
| Q&A over internal documents | RAG |
| Knowledge base that changes frequently | RAG |
| Need citations and audit trail | RAG |
| Using a proprietary frontier model | RAG |
| Need consistent output format | Fine-tune |
| Domain with specialised vocabulary | Fine-tune (or RAG + fine-tune) |
| High call volume, cost sensitivity | Fine-tune smaller model |
| Sub-100ms latency requirement | Fine-tune + self-host |
| Both factual grounding and behaviour control | RAG + fine-tune |
Start with prompting before either RAG or fine-tuning
Before investing in RAG or fine-tuning, explore how far prompt engineering alone can take you. Many use cases can be solved with a well-designed system prompt. RAG and fine-tuning should address limitations of prompting, not be your first tool.
LLM strategy and AI platform architecture are areas I work on extensively with clients. If you're planning an enterprise AI deployment and need help with the architecture decisions, let's talk.