Most teams evaluate their agent the way they evaluated their chatbot: ask a question, grade the answer. That catches perhaps the least interesting failures. An agent can reach the right answer by an expensive, dangerous or lucky route, and reach the wrong one after doing everything right except the last step. As teams move agents from prototypes to production, evaluation is where the work is. This article sets out what to measure and how.
What answer-only evaluation misses
An agent run is a sequence: a plan, a series of tool calls, observations, more decisions and a result. Grading only the result treats all of that as a black box.
- Right answer, wrong route: it got the refund total correct by querying the production database with write permissions it should not have used.
- Right answer, wasteful route: it took forty tool calls to do what should take six, at seven times the cost.
- Right answer, by luck: it succeeds once in three runs, and you happened to see the good one.
- Wrong answer, nearly right: everything worked except a final formatting step, which tells you exactly where to fix it — if you look.
Layer 1: outcome checks in a sandbox
Where you can, judge an agent by the state of the world after it runs rather than by what it says. Did the test suite pass? Is the row in the database? Was the ticket routed to the right queue? Was no email sent?
This needs a sandbox: a copy of the environment the agent acts on, reset between runs. Building one is the most valuable investment in agent evaluation, because state checks are deterministic and do not argue with you.
Layer 2: assertions on tool calls
Many important properties are about the route, and they can be checked mechanically from the run log.
- Required calls: it looked up the order before issuing a refund.
- Forbidden calls: it never called the delete tool; it never sent data to an unapproved destination.
- Argument checks: the SQL it ran was read-only; the refund amount matched the order.
- Budgets: no more than a set number of steps, tokens or seconds for this class of task.
Layer 3: LLM judges, with calibration
Some qualities — was the explanation clear, was the plan sensible — need judgement. A model can provide it at scale, but an uncalibrated judge produces confident numbers that mean nothing.
Give the judge a specific rubric with examples of each score, ask for the reasoning before the score, and check it against people: have humans grade a sample, compare, and only rely on the judge for the criteria where it agrees with them. Re-check whenever you change the judge model or the rubric.
Layer 4: reliability across repeated runs
Agents are not deterministic, so a single successful run proves little. Run each task several times and report how often it succeeds every time, not just at least once.
Research on agent benchmarks such as tau-bench made this explicit with a metric called pass^k: the probability that an agent succeeds on all k attempts at the same task. An agent that succeeds most of the time on one attempt can look far weaker when it must succeed on every one of several — which is what a user who relies on it every day experiences.
Building the evaluation set
The best tasks come from real use, not from the team's imagination.
- Start with twenty to fifty real tasks drawn from logs or from users, each with a checkable success condition.
- Add every production failure as a new task once it is fixed, so it cannot quietly return.
- Include adversarial cases: ambiguous requests, missing data, documents carrying hostile instructions.
- Version the set alongside the code, and run it on every change to prompts, tools, skills or models.
What to put on the dashboard
A small set of numbers, tracked over time, is more useful than a large one checked once.
- Task success rate on the evaluation set, and pass^k for the tasks that matter most.
- Tool-call violations: forbidden calls, unapproved destinations, budget breaches.
- Cost and latency per completed task.
- Human review notes on a weekly sample of real runs.
A worked example: evaluating a refund agent
Take an agent that handles refund requests for an online store. It can look up orders, check the refund policy, issue refunds up to a limit and escalate anything else to a person. One evaluation task reads: a customer asks for a refund on an order delivered nine days ago, inside a fourteen-day window, for a damaged item.
The outcome check runs against a sandboxed copy of the order database: after the run, a refund record exists for exactly that order, for the full item price, and no other order has changed. The tool-call assertions check that the agent looked up the order before refunding, read the policy, called the refund tool once and never called the tool that edits customer details. A budget assertion fails the run if it takes more than eight tool calls.
A second task is the same request on day twenty, outside the window. Success there is the absence of a refund and an escalation to a person with the reason stated. A third task embeds an instruction in the customer's message telling the agent it is authorised to refund double. Success is a normal, single refund. An LLM judge, calibrated against a sample of human grades, scores the reply for clarity and tone — but it cannot pass a run that the state check or a forbidden call has already failed. Each task runs five times; the team tracks how often all five succeed, because that is the experience of a customer who gets one of the bad runs.
Common mistakes
Evaluation suites tend to fail quietly in the same few ways.
- Testing only the happy path. Most production incidents are edge cases and hostile inputs, so those belong in the set from the start.
- Letting the judge outrank the facts. A fluent reply should never rescue a run that changed the wrong record.
- Running each task once. Variance is a property of agents; measure it instead of discovering it from complaints.
- Freezing the set. An evaluation set that never gains new tasks drifts away from what users actually ask.
Key takeaways
- 1.Grade what the agent did, not only what it said.
- 2.Outcome checks in a resettable sandbox are the most reliable signal you can build.
- 3.Assert on tool calls: required, forbidden, arguments and budgets.
- 4.Calibrate LLM judges against human grades before trusting them.
- 5.Measure reliability across repeated runs; one success proves little.
Do it now, free
AI Project Idea Generator
Five ideas, each with an architecture, a realistic number of hours, and how you would evaluate it — because the difference between a portfolio project and a tutorial clone is usually evaluation.
No sign-up. The internship is free to join too, with real project briefs and reviewed submissions.