Trusting the numbers: non-determinism, environments, and evals in CI

Run the same eval twice on the same AI agent, without changing anything, and the pass rate can move by several points. If the difference between two versions of the agent is smaller than that, the eval hasn't told you which one is better. This article is about what makes the numbers of an agent eval trustworthy: running each case more than once, giving the agent a world it can't damage, and running evals where they catch problems in time.

An eval is a set of cases, each a task with an expected outcome, that the agent runs and graders turn into results, so that two versions can be compared instead of guessed at. I described why it is needed in Without evals, every change is a guess, and how to write the cases and choose between code, model and human graders in Building an eval for an agent.

The agent

The example is a customer-support agent for an online shop. It reads a ticket, such as "my order arrived broken, I want my money back", and has four tools: lookup_order, check_refund_policy, issue_refund and reply_to_customer. The policy gives a full refund within 30 days of delivery, store credit after that, and sends anything above €200 to a person. It sometimes refunds outside the policy, asks for information already in the ticket, refunds the same order twice, or replies politely without doing anything.

One run proves little

An agent doesn't give the same result every time it runs the same case. The model samples its answers, so two runs can choose different tools, phrase a question differently or stop at a different point. Setting the temperature to zero reduces this but doesn't remove it, and the results of tools and the timing of calls vary anyway.

So I run each case several times. Here are four cases, five runs each:

The average pass rate is 85%. Two other numbers describe the same table in more useful ways.

pass@k is the share of cases where at least one of k runs passed. Here pass@5 is 1.00: every case succeeds at least sometimes. This is the right measure when someone can pick the good attempt out of several, for example when a coding agent's attempts are checked by a test suite and only one has to work.

pass^k is the share of cases where all k runs passed. Here pass^5 is 0.50: half of the cases fail at least once in five tries. This measure was introduced with τ-bench, a benchmark of customer-service agents, and it fits them for a simple reason: each customer gets one run, and nobody picks the good one. An agent that escalates a €340 refund in three runs out of five refunds it without asking anyone for two customers in five.

The same goes for comparing versions. Before reading a difference between two versions, I run the current version three to five times and look at how much its pass rate moves on its own. A change that stays within that spread is not evidence of anything, however good it looks.

A world the agent can't break

Every run has to start from the state the case describes and end without touching anything real. For the support agent that means the four tools work on a test store: lookup_order reads orders the case put there, issue_refund marks an order as refunded instead of moving money, and the graders read the same store when the run ends. Each run gets a fresh copy, so a refund in one run can't change the outcome of the next.

These fake tools are what make an eval cheap and repeatable, but they can drift from the real ones. If the real lookup_order starts returning dates in another format and the fake doesn't, the eval keeps passing while production fails. I keep a few tests that call the real tools in a staging environment and check that they still return what the fakes return.

Repeatable also means knowing what was run. Next to each result I record the model and its version, the prompt version, the tool versions, the judge's model and rubric version, and how many runs each case had. Without that, a change in the numbers a month later can't be explained.

Offline and online

Everything so far is an offline eval: fixed cases, run before a change goes out. It can only find the problems someone wrote a case for.

Production traffic has the problems nobody wrote a case for, but there is no expected outcome to compare with. What can be measured are signals: the share of tickets escalated, refunds per ticket, tickets reopened by the customer within a week. Some of the graders from the offline eval also work here unchanged. The invariants on the trajectory, like issue_refund called at most once per order or never above €200, can be checked on every production transcript, and a sample of replies can go to the same judge or to a person.

The two are a loop. A ticket that went wrong in production becomes a new offline case, with the outcome it should have had, and from then on every version is checked against it.

Evals in CI

Once there is an eval, it should run on every change to the prompt, the tools or the model, the same way tests run on every change to the code. I split the cases into two suites with different jobs.

The regression suite holds the cases the agent already passes reliably. It must stay green: a change that breaks one of them, or violates an invariant in any run, doesn't get merged. It answers the question "what did this change break?".

The capability suite holds the cases the agent fails today, the things I want it to learn. It is expected to be mostly red, and nothing is blocked when it is. Its pass rate measures progress. When a case in it passes in every run, measured with pass^k rather than once, it moves to the regression suite.

This also suggests an order of work that will be familiar from test-driven development: before changing the prompt to fix a failure, write the case that shows the failure, see it fail, then change the prompt until it passes without breaking the regression suite.

Running every case five times on every commit costs money and time. I run a smaller number of runs per case on each pull request and the full suite before a release or every night. The invariants are cheap, so they run on everything.

What this doesn't tell you

Public benchmarks, like SWE-bench for coding agents or τ-bench for support agents, measure models on someone else's tasks. They help choose a model to start from, but they don't say whether your agent handles your tickets. And since their tasks are public, a model may have seen them during training, which makes its score look better than its ability.

Cost and latency are real measures of an agent, and they belong in the same report as the pass rate: a version that passes one more case but doubles the cost per ticket is not obviously better. I left them out here because they need their own discussion.

Finally, none of the cases in this series is written by someone trying to make the agent misbehave. A ticket that contains instructions addressed to the agent is a different kind of case, and the problem behind it is the one I described in Content is not an instruction and Who judges the input. LLM Application Engineering covers testing and evals, and those defences, as parts of the same application.