Run a coding agent twenty times on the same task and you get several different outcomes. Reliability lives in the shape of that distribution, and specs, hooks and retries reshape it. A way to read the shape, a lens on what interventions do to it, and 400 runs of Claude Code on a small Laravel app.
Why one run of an agent eval proves little, what pass@k and pass^k measure, how to give the agent a world it cannot damage, and how evals fit next to production monitoring and continuous integration.
Where the cases of an agent eval come from, when a check should be code, a model or a person, and why the outcome of a run matters more than the path the agent took to reach it.
Why an AI agent needs evals before its next prompt or model change, what makes an agent harder to evaluate than a single LLM call, and the task, run and grade loop every eval is built on.