Practical guides
How to evaluate an AI agent
A model produces an answer you can mark. An agent takes a sequence of actions, some of them irreversible. Marking the final answer misses most of what matters.
Agent evaluation is where most teams discover that their testing approach was built for a different kind of system. A question-and-answer model has one output per input, and a held-out set with agreed answers works well. An agent chooses tools, takes several steps, changes state in other systems, and can arrive at a correct answer through a route you would never have approved.
Which means correctness on the final answer is necessary and nowhere near sufficient. What follows is what to measure instead.
Four things worth measuring
Success rate is one number that hides four. Route, restraint, and the cost of being wrong are the other three.
Building a suite that holds up
The evaluation set is the durable asset here. Models will change several times; the definition of good should not have to.
Real cases first
Draw from production history, including the ones that went wrong. Synthetic cases are useful for coverage and useless for calibration.
Include the unhappy path
Missing fields, contradictory records, requests outside policy, and the case that arrives twice. This is where agents differ most from each other.
Include cases that must be refused
Somewhere between a tenth and a fifth of the suite. Without them you cannot detect an agent that has become agreeable.
Freeze a portion
Hold back a slice that nobody optimises against and that is never shown to the system. It is the only defence against quietly fitting the test.
Public benchmarks such as Stanford's HELM are useful for orientation and for nothing else. They cannot contain your policies, your exceptions, or your data. Anyone selling on leaderboard position is answering a question you did not ask, which is the argument in how to choose a model for production.
You need somewhere safe for it to act
The part teams underestimate: evaluating an agent requires an environment where its actions are real enough to be meaningful and contained enough to be harmless.
A sandbox with real shapes
Copies of the systems the agent touches, with realistic data volumes and realistic latency. An agent that has only met a fast, tidy mock will behave differently in production.
Deterministic replay
The same case, runnable repeatedly, with tool responses recorded. Without it you cannot tell a fixed bug from a lucky sample.
Tool permissions as a test surface
What the agent is allowed to call is part of what you are evaluating. The OWASP Top 10 for LLM applications is a reasonable starting list of what to probe.
Cost and latency captured per run
Recorded alongside quality, not in a separate report. Cost per completed task is the number that decides whether an agent reaches production.
Guidance on managing this class of risk is set out in the NIST AI Risk Management Framework, and our own position is in responsible AI.
Evaluation does not stop at launch
An agent that passed on Tuesday can fail in September without anything visible changing: a model updated, a tool's response format shifted, the population of requests drifted.
So the suite runs on a schedule, not once. Track success rate, refusal rate, average steps, and cost per task as a time series, and alert on the second derivative rather than the level. A refusal rate that halves in a week is a stronger signal than an absolute number, and it usually arrives before the complaints.
Sample live traffic for human review at a fixed rate, permanently. The cases nobody looks at are where the expensive failures accumulate, and they are the ones that turn into the failure that presents as success.
On using a model as the judge
Automated grading by another model is practical and worth doing, with two conditions. Calibrate it against human labels on a subset and report the agreement rate, so you know the error bar on your error bar. And never let the judge be the same model as the one being judged, on the same prompt, in the same run.
Human review stays in the loop for the irreversible cases and for anything the agent refused. Those are the two populations where automated judgement is least reliable and the stakes are highest.
None of this is exotic. It is ordinary quality engineering applied to a system that acts. The reason it gets skipped is that an agent demonstrates well long before it is dependable, and a demonstration is much easier to arrange than a suite. This is the discipline behind our model evaluation work.
If it has not been evaluated, it has been demonstrated.
Building the suite is usually a fortnight of work and it outlives every model you will use. We do this as its own engagement.