tags /
Testing
Every agent run is a sample
Run a coding agent twenty times on the same task and you get several different outcomes. Reliability lives in the shape of that distribution, and specs, hooks and retries reshape it. A way to read the shape, a lens on what interventions do to it, and 400 runs of Claude Code on a small Laravel app.
Trusting the numbers: non-determinism, environments, and evals in CI
Why one run of an agent eval proves little, what pass@k and pass^k measure, how to give the agent a world it cannot damage, and how evals fit next to production monitoring and continuous integration.
Five reasons to write a test, with Pest 5
TDD, regression, characterization, acceptance and smoke tests can look identical in a Pest file. What separates them is the reason they were written, and that reason decides how you read them, name them, and whether you can delete them. A field guide with one running example in Pest 5.
Building an eval for an agent: cases, graders, and what to grade
Where the cases of an agent eval come from, when a check should be code, a model or a person, and why the outcome of a run matters more than the path the agent took to reach it.
Without evals, every change is a guess
Why an AI agent needs evals before its next prompt or model change, what makes an agent harder to evaluate than a single LLM call, and the task, run and grade loop every eval is built on.