Teams ship RAG systems on vibes and then wonder why quality drifts. Evaluation is what turns 'it seems better' into a number you can defend to a stakeholder and regress against on every change. The key insight is that a RAG system has two engines — retrieval and generation — and you must measure them separately to know which one is failing.
Why separate retrieval from generation
A RAG answer can be wrong for two very different reasons: the retriever failed to find the relevant evidence, or the retriever found it and the generator ignored, misused, or contradicted it. A single end-to-end accuracy score cannot tell these apart, so it cannot tell you what to fix.
Evaluate each engine on its own axis. Retrieval metrics ask 'did we find the right context?' Generation metrics ask 'given the context, did we produce a good, grounded answer?' This decomposition is the single most useful move in RAG evaluation.
Retrieval metrics that matter
For retrieval you need a labeled set: queries paired with the documents that should be retrieved. With that, measure context recall (did we retrieve all the relevant chunks?) and context precision (are the retrieved chunks actually relevant, and are the relevant ones ranked near the top?). Recall bounds how good your answer can possibly be; if the evidence is not retrieved, no model can use it.
Rank-aware metrics like MRR and nDCG capture whether relevant chunks land near the top, which matters because generation quality degrades when the signal is buried among distractors. Build this labeled set from real user queries — a few hundred well-chosen examples beat a synthetic benchmark that does not reflect your corpus.
- Context recall: fraction of the relevant chunks that were retrieved — your quality ceiling.
- Context precision: are retrieved chunks relevant and ranked highly, not padded with noise.
- MRR / nDCG: rank-aware quality of the retrieved set.
Generation metrics that matter
For generation, the metric that matters most in the enterprise is faithfulness (also called groundedness): is every claim in the answer supported by the retrieved context? A faithful answer that says 'I do not have that information' is safer than an unfaithful one that fabricates a confident number. Alongside it, measure answer relevance (does the answer address the actual question?) and, where you have references, correctness against ground truth.
Frameworks like RAGAS operationalize these — faithfulness, answer relevance, context precision and recall — using LLM-as-judge scoring so you can run them at scale. Treat citation accuracy as a first-class check too: verify that the sources cited actually support the claims, not just that citations exist.
- Faithfulness / groundedness: every claim traceable to retrieved context — the top enterprise metric.
- Answer relevance: the response actually addresses the question asked.
- Correctness: agreement with ground-truth answers where you have them.
- Citation accuracy: cited sources genuinely support the stated claims.
LLM-as-judge, used responsibly
Human evaluation is the gold standard but does not scale to every commit. LLM-as-judge — using a strong model to score outputs against a rubric — scales well and correlates reasonably with human judgment when done carefully. The discipline is in the rubric: give the judge explicit, narrow criteria and few-shot examples rather than a vague 'is this good?'
Guard against the known biases. Judges favor longer and more confident answers and can prefer outputs from their own model family. Calibrate the judge against a human-labeled sample periodically, and keep a small human-graded golden set as ground truth for your automated pipeline. The judge accelerates evaluation; it does not replace human oversight entirely.
Build a regression suite, not a one-off report
Evaluation is only valuable if it runs on every change. Assemble a fixed dataset of representative and adversarial cases — including the hard queries that have failed before — and run the full metric suite automatically whenever you change the model, prompt, chunking, or retrieval configuration. This catches the regression where fixing one query quietly breaks five others.
Wire it into CI so a prompt tweak that tanks faithfulness fails the build rather than reaching users. In our experience, the existence of a trustworthy regression suite is what lets teams iterate on RAG quickly without fear, which is worth more than any single clever retrieval trick.
Close the loop with production signals
Offline evaluation is necessary but not sufficient. Instrument production for the signals users give you: thumbs up/down, escalation to a human, query reformulation (a sign the first answer failed), and 'not found' rates. Feed failing production queries back into your evaluation set so your test corpus tracks reality.
The mature setup is a virtuous cycle: production surfaces real failures, those failures become regression cases, and the regression suite prevents them from recurring. That loop, more than any individual metric, is what keeps a RAG system improving over time instead of quietly degrading.
Key takeaways
- 1.Evaluate retrieval and generation separately — a single accuracy score hides which engine failed.
- 2.For retrieval, measure context recall (your quality ceiling), precision, and rank-aware quality.
- 3.For generation, faithfulness/groundedness is the top enterprise metric, alongside relevance and citation accuracy.
- 4.Use LLM-as-judge for scale with careful rubrics, but calibrate against a human-graded golden set.
- 5.Build a regression suite in CI and feed production failures back into it to keep improving.