Retrieval-Augmented Generation — RAG — has become the dominant pattern for connecting LLMs to business data. If you have asked an AI assistant a question and it answered using content from a specific document, product catalogue, or knowledge base, you have used a RAG system. But "RAG" is used to describe everything from a two-line Python script to a sophisticated multi-stage retrieval pipeline, and the gap between those two implementations is enormous.
How RAG Actually Works
The core RAG pattern has three steps: retrieve, augment, generate. When a user asks a question, the system retrieves relevant documents from a knowledge store (typically using vector similarity search), augments the user's question with those documents as additional context, and then sends the augmented prompt to an LLM to generate a grounded answer.
What makes this powerful is that the LLM does not need to "know" your data in advance. Instead of fine-tuning a model on your documents (expensive, slow, produces stale knowledge), you retrieve the relevant documents at query time and let the model reason over them. The model's general language capability is combined with your specific, up-to-date data.
- Documents are chunked and converted into vector embeddings — numerical representations of semantic meaning
- Embeddings are stored in a vector database (Pinecone, Weaviate, pgvector, Qdrant)
- At query time, the user's question is also embedded and compared against stored vectors
- The most semantically similar chunks are retrieved and injected into the prompt as context
- The LLM generates an answer grounded in the retrieved content
When RAG Is the Right Choice
RAG is the right pattern when your use case involves answering questions over a corpus of documents that changes over time, is proprietary to your organisation, or is too large to fit in a single context window.
- Internal knowledge bases: policy documents, SOPs, product documentation
- Customer support: answering questions over product manuals, FAQs, and past ticket resolutions
- Legal and compliance: querying contracts, regulations, case precedents
- Research: synthesising information across large document collections
- Sales enablement: answering questions over pricing, product specs, competitor analyses
When RAG Is Not the Right Choice
RAG is not the right choice when your use case requires reasoning over structured data (use SQL or a structured API instead), when the documents are few enough to fit in a single context window (just include them directly), or when response latency is a hard constraint (retrieval adds round-trip time).
Why RAG Systems Fail in Production
Most RAG implementations that fail do so for one of three reasons: poor chunking, poor retrieval, or poor context injection. These are engineering problems, not model problems.
Poor chunking
Splitting documents naively — by character count or fixed line breaks — creates chunks that cut sentences mid-thought or separate a question from its answer. Good chunking respects document structure: paragraphs, headings, and semantic boundaries. For long documents, overlapping chunks (each chunk shares N tokens with the previous) improve recall significantly.
Poor retrieval
Vector similarity alone often misses relevant content when terminology varies or when the question is vague. Production RAG systems typically use hybrid retrieval: combining vector search (semantic) with keyword search (BM25) and re-ranking the combined results with a cross-encoder model. This hybrid approach meaningfully outperforms pure vector search on real-world queries.
Poor context injection
Dumping retrieved chunks into the prompt without structure confuses the model. Good context injection includes metadata (source document, section heading, date), organises chunks in logical order, and uses explicit delimiters so the model knows where context starts and ends. Including a citation instruction ("cite the source document for each claim") dramatically improves output verifiability.
What Production RAG Actually Requires
A prototype RAG system can be built in an afternoon. A production RAG system requires infrastructure that the prototype does not have.
- Document ingestion pipeline: automated processing of new and updated documents, with re-embedding on change
- Chunking strategy: tested against real queries, not assumed from defaults
- Hybrid retrieval: vector + keyword + re-ranking for consistent recall across query types
- Metadata filtering: scope retrieval to relevant document subsets (by date, department, access level)
- Citation and source attribution: every output should link to the source chunks it was grounded in
- Evaluation framework: automated relevance and faithfulness scoring, run on a schedule
- Access control: ensure retrieval respects document-level permissions