A RAG demo is the easiest impressive thing in AI right now. Forty lines of Python, a curated PDF corpus, a GPT-4o call, and you have something that looks, in a conference room, like it can answer any question from your internal knowledge base. The board loves it. The CEO wants it shipped in six weeks. And then reality arrives.
I have been called into three different post-mortems in the past fourteen months for systems that followed exactly this arc. Stunning demo. Dismal production. In each case, the failure was not the model. The model was fine. The failure was everything the demo hid: data quality, retrieval mechanics, latency under load, hallucination rates on real queries, and the complete absence of observability once the system went live.
The good news is that production-grade RAG is buildable. The bad news is that it looks almost nothing like the demo.
Why the Demo Always Lies
The demo RAG system has three invisible advantages that vanish the moment real users show up.
The corpus is clean. You built the demo with ten polished PDFs, a clean web scrape, or a curated subset of your knowledge base. Production means ingesting 40,000 Confluence pages last updated in 2019, SharePoint documents in six languages, PDFs scanned from paper with inconsistent OCR quality, and Slack exports where the signal-to-noise ratio is around 10%.
The queries are cooperative. You demoed with queries you wrote yourself, designed to retrieve cleanly. Real users ask fragmented questions, use internal jargon that isn't in the documents, misspell product names, and ask multi-part questions that require synthesising across three different document sections that happen to be in different language registers.
There is no feedback loop. In the demo, you see the answer, nod approvingly, and move on. In production, you need to know which queries returned irrelevant context, which answers contained hallucinated details, and which retrieval paths consistently fail. Without instrumentation, you are flying completely blind.
The Four Failure Modes I See Repeatedly
Failure 1: Chunking That Destroys Context
Most RAG tutorials split documents into fixed-size chunks — 512 tokens, 1,024 tokens, whatever feels right — with or without overlap. This works well for clean, structured documents. It fails systematically for everything else.
A contract clause that spans a page break gets split in the middle. A table gets fragmented so that the header row and the data rows end up in different chunks. A policy document where the definition in section 2 is essential to interpreting the requirement in section 7 returns only one of the two when either is queried.
What actually works: semantic chunking that splits at natural document boundaries (paragraphs, sections, table rows), combined with parent-document retrieval where the embedding represents a small chunk but the context returned to the model includes the surrounding parent section. This dramatically improves coherence at the cost of higher complexity and a more carefully designed ingestion pipeline.
Failure 2: Naive Top-K Retrieval
Vector similarity search is not the same as relevance. A query about "our refund policy for enterprise customers" will retrieve chunks that are semantically adjacent — all customer policy content — but not necessarily the specific clause that applies to enterprise accounts as opposed to SMB.
The naive top-5 cosine similarity result set is often 60–80% irrelevant in production on a real knowledge base. The model then tries to answer from a context window full of loosely related content and either hedges into uselessness or confidently synthesises an answer from the wrong source material.
What actually works: hybrid retrieval combining dense vector search with sparse BM25 keyword search, followed by a reranker (a cross-encoder model that scores retrieved chunk-query pairs for actual relevance before passing context to the LLM). The reranker step alone typically moves answer quality from "sometimes useful" to "reliable enough to trust." It adds 50–150ms of latency. It is worth it.
Failure 3: No Query Understanding Layer
Users do not query RAG systems the way you think they will. They ask conversational follow-up questions that reference earlier turns ("what about for the UK version?"), they ask vague questions that require decomposition ("how does our onboarding work?"), and they ask questions that combine a lookup with a calculation ("what's the average deal size in the Q3 report and how does that compare to Q2?").
A single-turn retrieval system treats every query in isolation. It has no memory of what the user asked thirty seconds ago. It cannot decompose a compound question into parallel sub-queries and synthesise the results. It retrieves for "how does our onboarding work" and gets back a random slice of onboarding documentation that doesn't actually answer the question.
What actually works: a query rewriting layer that uses a fast, cheap model to classify the query type, expand it with synonyms and related terms, and — for conversational interfaces — reformulate it as a standalone query that doesn't depend on prior context. For complex questions, HyDE (Hypothetical Document Embeddings — generate a hypothetical answer, embed it, and use that for retrieval) outperforms direct query embedding by a meaningful margin on enterprise knowledge bases.
Failure 4: No Hallucination Guard at the Output Layer
RAG reduces hallucination relative to pure generation. It does not eliminate it. A model with a context window full of partially relevant chunks will still confabulate — filling in gaps in the retrieved evidence with plausible-sounding but fabricated details. In a legal, compliance, financial, or HR context, this is not an acceptable failure rate.
The demo never shows you this because the curated corpus covers the query cleanly. In production, the coverage gaps in your knowledge base are exactly where the model starts making things up.
What actually works: citation grounding where every claim in the model's output must be directly supported by a retrieved chunk (with source reference), combined with a faithfulness check — a second LLM call or a fine-tuned classifier that scores the output against the retrieved context and flags or suppresses answers that introduce content not present in the sources. This is not free. It adds cost and latency. But for any use case where the answer has real consequences, it is not optional.
The confidence problem
RAG systems fail in a particularly dangerous way: they fail confidently. A system that says "I don't know" is manageable. A system that says "According to your internal policy, the answer is X" — where X is subtly wrong, or right for a different context — erodes trust in ways that are very hard to recover from. The failure mode you are really guarding against is not "the system doesn't answer." It is "the system answers wrongly and the user acts on it."
What Production-Grade RAG Actually Looks Like
Here is the architecture gap between what gets demoed and what gets deployed successfully. I use this as a review checklist when I inherit or assess an existing RAG implementation.
| Layer | Demo Approach | Production Requirement |
|---|---|---|
| Ingestion | Manual upload of clean files | Automated pipeline with format handling, OCR, deduplication, versioning |
| Chunking | Fixed-size token splits | Semantic chunking with parent retrieval, table-aware parsing |
| Embedding | Single embedding model | Domain-adapted or fine-tuned embeddings, multi-lingual if needed |
| Retrieval | Dense vector top-k | Hybrid dense + sparse, reranker, query expansion |
| Query handling | Single-turn direct retrieval | Query rewriting, HyDE, multi-step decomposition for compound queries |
| Output guard | None | Citation grounding, faithfulness scoring, source attribution in UI |
| Observability | None | Retrieval trace logging, answer quality scoring, query failure dashboard |
| Freshness | Manual re-ingestion | Incremental indexing, document update detection, version-controlled embeddings |
| Latency SLA | Not defined | P95 target, caching layer for frequent queries, async for complex chains |
The full production pipeline, from raw document to trusted answer, flows through each of these layers in sequence:
The Observability Gap Is the Most Dangerous One
I put observability last in the table but it is the failure mode that compounds every other one. Without logging what was retrieved, what context was passed to the model, and what the model returned — alongside a mechanism to collect user feedback — you have no idea which failure modes are occurring at what rate.
In every post-mortem I have run on a failed RAG system, the team could tell me the demo worked great. None of them could tell me what the retrieval precision was on real production queries, which document sections were systematically failing to surface, or what percentage of answers contained unsupported claims. They found out through user complaints, which is the worst possible feedback loop.
The instrumentation I build from day one: structured logging of every retrieval event (query, retrieved chunk IDs, similarity scores, reranker scores), every generation event (context length, model, latency, output), and a simple feedback mechanism (thumbs up/down or "was this answer correct?"). This data feeds a weekly quality review and drives the prioritisation of which failure modes to fix first.
The question to ask before you build
Before committing to a RAG build, ask: what is the coverage and quality of the source documents? If the answer is "we have a lot of documents but they're not consistently maintained or structured," the AI problem is actually a data governance problem. RAG is a retrieval system. It cannot retrieve what isn't there, and it cannot reliably surface structure that doesn't exist. Fixing the documents is often more valuable than optimising the retrieval pipeline.
The Build Sequence That Actually Works
Spend one week understanding what is actually in the knowledge base. What formats, what languages, what update frequency, what coverage gaps. The corpus audit determines the ingestion architecture, not the other way around.
Curate 100–200 real production-representative queries with ground-truth answers before you build anything. This is your evaluation set. Every architectural decision gets validated against it. If you cannot get ground-truth answers, your users do not have a clear enough use case to build for yet.
Start with the simplest retrievable system — direct embedding, top-5 cosine, no reranker. Measure retrieval precision@5 and answer faithfulness on your eval set. Now you have a baseline. Every improvement (hybrid retrieval, reranker, query rewriting) should be validated against it. This prevents complexity for its own sake.
The logging and feedback infrastructure is not a post-launch feature. It ships with v1. Otherwise you are operating blind from day one.
Define the minimum acceptable retrieval precision and hallucination rate for production use. Do not ship until those thresholds are met on the eval set. This forces the quality conversation before users experience the failures, rather than after.
The Conversation I Have With Every Leadership Team
When a CEO tells me the demo looked great and they want it live in six weeks, I say: "The demo is a proof of concept. It proves the technology works. It does not prove your data is ready, your failure modes are understood, or that the system behaves reliably under real query conditions."
Six weeks is sometimes possible — if the corpus is clean, the use case is narrow, the stakes are low (internal productivity tool, not customer-facing), and the team has RAG production experience. For most enterprise knowledge-base assistants, twelve to sixteen weeks is closer to realistic for a version that can be trusted.
The organisations that get this right ship more slowly and more confidently. The ones that rush the demo to production spend the following six months trying to rebuild trust with users who have already decided the system is unreliable. That trust is very hard to recover once lost.
If you are evaluating an AI knowledge-base assistant or have a RAG implementation that is underperforming in production, let's talk — book a 30-minute discovery call and we will diagnose what is actually breaking and what it would take to fix it.