Enterprise RAG: Beyond Simple Document Q&A

Why naive top-k retrieval fails

Pure vector similarity is great at fuzzy semantic matching and terrible at exact terms — part numbers, error codes, policy IDs, acronyms your business invented. It also has no notion of recency or authority, so a superseded 2021 policy scores the same as this quarter's approved version. And it happily retrieves documents the current user is not allowed to see.

In enterprise settings these are not edge cases; they are the majority of real questions. The fix is not a bigger model. It is a retrieval pipeline that treats search as a first-class engineering problem.

Hybrid retrieval: dense plus sparse

Combine dense vector search with sparse keyword search (BM25 or its variants) and fuse the results, typically with reciprocal rank fusion. Dense retrieval catches paraphrases and concepts; sparse retrieval nails exact identifiers and rare terms. Together they cover each other's blind spots.

pgvector on Postgres is an excellent default when you already run Postgres — you get transactional consistency, joins to your metadata, and one fewer system to operate. Reach for a dedicated store like Pinecone, Weaviate, or Qdrant when scale, latency, or advanced filtering justify the extra moving part.

• Run dense and sparse retrieval in parallel and fuse ranks rather than scores, which avoids scale-normalization headaches.
• Store rich metadata alongside vectors: source, author, effective date, document status, and sensitivity label.
• Benchmark recall@k on your own labeled queries — public benchmarks rarely predict performance on your corpus.

Rerank before you generate

First-stage retrieval optimizes for recall: cast a wide net, pull maybe 50 candidates. Then use a cross-encoder reranker (Cohere Rerank, or an open model like bge-reranker) to score query-document pairs directly and keep the top 5 to 8. Cross-encoders are slower per pair but dramatically more precise than the bi-encoder embeddings used in first-stage search.

This two-stage design is the single highest-ROI change we make to underperforming RAG systems. It consistently lifts answer quality more than swapping the generation model, and it lets you feed the LLM fewer, better chunks — which cuts both cost and hallucination.

Chunking and metadata are the real work

Fixed-size character chunks split tables mid-row and sever headings from their content. Chunk along document structure instead — sections, headings, table boundaries — and attach the heading path to each chunk so the model has context about where the text lives. For long documents, a parent-document strategy (retrieve small precise chunks, then expand to their parent section for generation) balances precision and context.

Every chunk should carry metadata that powers filtering: department, product line, effective date, document status. This is what lets you answer 'according to the current policy' by filtering to status = active before you ever rank.

Security: retrieval must respect permissions

This is where most enterprise RAG projects meet legal. If the index contains HR records, contracts, and board material, retrieval must filter by the requesting user's entitlements at query time — not after generation, when the sensitive text is already in the prompt and possibly in your logs.

Store access-control metadata (owning group, ACL tags) on every chunk and inject a mandatory metadata filter derived from the user's identity into every query. Log which documents were retrieved for which user so you can answer an audit question months later. On regulated data, keep embeddings and inference inside your compliance boundary — Azure OpenAI or Amazon Bedrock with a private endpoint rather than a public API.

• Enforce entitlement filters in the retrieval layer, never in a post-hoc prompt instruction the model might ignore.
• Redact or exclude PII at ingestion when the use case does not require it.
• Keep a retrieval audit log tying user, query, and returned documents together.

Keep the index fresh

Enterprise knowledge changes daily. Build an incremental ingestion pipeline that detects changed source documents, re-chunks and re-embeds only what moved, and marks superseded versions as inactive rather than deleting them outright. Stale answers erode trust faster than occasional 'I do not know' responses.

Wire ingestion to your source systems' change events where you can — SharePoint, Confluence, a document management system — so freshness is a property of the architecture, not a nightly cron job someone forgets to monitor.

Measure grounding, not vibes

Track retrieval metrics (recall@k, reranker precision) separately from generation metrics (faithfulness to retrieved context, answer relevance, citation accuracy). When quality drops, this separation tells you instantly whether the retriever failed to find the evidence or the generator ignored it.

Require citations in every answer and verify they actually support the claim. A RAG system that cites its sources and occasionally says 'not found in the knowledge base' is far more valuable to an enterprise than a confident one that quietly makes things up.