A RAG Evaluation Harness
Measure retrieval and generation separately, so you know which one to fix.
Intermediate8–14 hours estimatedEvaluation
The problem
Most RAG systems are judged by asking a few questions and reading the answers, which detects catastrophe and nothing subtler. Without separate measurement you cannot tell whether a bad answer came from bad retrieval or bad generation, and those have opposite fixes.
Who uses it: Any team with a RAG system in production and no way to tell whether a change made it better.
Before you start
- An existing RAG pipeline, even a rough one
- Python
Stack
Pythonpytest or a plain runnerYour existing retrieverA judging LLM