Skip to content

Series

Evaluating AI agents

How to tell whether a change made your agent better or worse: why agents need evals, how to build cases and graders for one, and how far to trust the numbers when every run can differ.

  1. Part 1

    Without evals, every change is a guess

    Why an AI agent needs evals before its next prompt or model change, what makes an agent harder to evaluate than a single LLM call, and the task, run and grade loop every eval is built on.

  2. Part 2

    Building an eval for an agent: cases, graders, and what to grade

    Where the cases of an agent eval come from, when a check should be code, a model or a person, and why the outcome of a run matters more than the path the agent took to reach it.

  3. Part 3

    Trusting the numbers: non-determinism, environments, and evals in CI

    Why one run of an agent eval proves little, what pass@k and pass^k measure, how to give the agent a world it cannot damage, and how evals fit next to production monitoring and continuous integration.