RAG
RAG is easy to demo on ten clean documents and hard the moment the corpus is real. The difficulty is retrieval, not generation.
Why the tutorial version stops working
Every RAG tutorial uses a handful of well-formed text files and produces something that looks like it works. Real corpora are scanned PDFs with two-column layouts, spreadsheets exported to text, near-duplicate policy documents where only the date differs, and a table whose meaning depends on a header eleven rows above it.
When quality drops on real data, the instinct is to change the model. It is almost never the model. The retriever returned three chunks, none of which contained the answer, and the model did what it was asked with what it was given. Before touching generation, look at what was actually retrieved.
Chunking is a judgement, not a setting
Fixed-size chunks with an overlap is the default because it is easy, not because it is right. It splits tables down the middle and separates a clause from the definition it depends on. Chunking on document structure — headings, sections, rows — keeps meaning intact, and costs an afternoon of parsing per document type.
The other half is what travels with the chunk. A chunk carrying its document title, section heading and date is retrievable by questions that never mention its text. A bare paragraph is not.
You cannot improve what you do not measure
Most RAG systems are evaluated by asking a few questions and reading the answers. That detects catastrophe and nothing subtler. Build a set of questions with known correct sources and measure retrieval separately from generation: did the right chunk come back at all, and did the answer use it.
Splitting those two numbers tells you where to spend. Bad retrieval and good generation means fix chunking or add re-ranking. Good retrieval and bad generation means fix the prompt or the grounding instruction. One combined score tells you neither.
Where this usually goes wrong
Reaching for a bigger model when retrieval is the problem
If the answer was not in the retrieved context, no model will produce it, and a larger one will produce a more convincing invention. Print what was retrieved before changing anything else.
Fixed-size chunking on structured documents
A 512-token window cuts tables in half and separates definitions from the clauses that use them. Chunk on the document’s own structure.
No re-ranking on a large corpus
Vector search is fast and approximate. Past a few thousand documents, retrieving twenty candidates and re-ranking them beats retrieving five and hoping, for a small latency cost.
Questions people actually ask
RAG or fine-tuning?
RAG for knowledge that changes or must be cited; fine-tuning for behaviour, format and tone. Facts belong in retrieval where they can be updated and attributed. A model fine-tuned on facts cannot tell you where an answer came from, and cannot be corrected without retraining.
Which vector database should I use?
If your data already lives in Postgres, pgvector removes an entire system from your architecture and is enough well past the point most projects reach. Dedicated stores earn their place at scale, with heavy filtering, or when you need features they specifically provide.
How do I stop it inventing answers?
Instruct it to answer only from the provided context and to say when the context is insufficient — then verify it actually does, because models comply inconsistently. Returning citations makes fabrication visible to the reader rather than hidden.
Build it, free
Project briefs with the assessment criteria published before you start, and written code review on what you submit. No payment to apply or to complete.
Read further
Enterprise RAG
Enterprise RAG: Beyond Simple Document Q&A
13 min read
Enterprise RAG
How to Evaluate a RAG System: Metrics That Matter
11 min read
Enterprise RAG
RAG vs Fine-Tuning: Which Does Your Enterprise Actually Need?
12 min read
Enterprise RAG
Vector Databases Compared: pgvector vs Pinecone vs Weaviate
13 min read
GenAI
How to Choose the Right LLM for Your Enterprise Use Case
12 min read