Evaluating AI agents
How to tell whether a change made your agent better or worse: why agents need evals, how to build cases and graders for one, and how far to trust the numbers when every run can differ.
-
Part 1
Without evals, every change is a guess
Why an AI agent needs evals before its next prompt or model change, what makes an agent harder to evaluate than a single LLM call, and the task, run and grade loop every eval is built on.
-
Part 2
Building an eval for an agent: cases, graders, and what to grade
Where the cases of an agent eval come from, when a check should be code, a model or a person, and why the outcome of a run matters more than the path the agent took to reach it.
-
Part 3
Trusting the numbers: non-determinism, environments, and evals in CI
Why one run of an agent eval proves little, what pass@k and pass^k measure, how to give the agent a world it cannot damage, and how evals fit next to production monitoring and continuous integration.