SYNTROX.ai
Blog

Regression Testing LLM Agents in CI: How to Catch What a Change Broke

5 min read

Write every known failure as a test case: a fixed input, the tools and context the agent had, and a check on the outcome rather than the exact wording. Run the suite on every model, prompt, tool and configuration change. Because outputs vary even at temperature 0, run each case several times, gate releases on pass rates per category, and count crashed or unscored runs as failures.

On this page

Suppose your agent has, at various points, invented a refund policy, emailed the wrong customer and followed an instruction hidden in a document. Each was fixed, probably by rewording a prompt.

The question a regression suite answers is whether those fixes are still holding after this week's changes. A red-team report goes stale unless its findings become tests that run on every release. This is the engineering side of that: how to build those tests so they mean something.

What a regression test for an LLM agent looks like

A test case for an agent has three parts.

  1. The input. The user's request, exactly as it arrived.
  2. The environment. Which tools were available and their descriptions, what each tool returned, which documents retrieval surfaced, the state of any records involved. Mock the tools with fixed responses, so the test exercises the agent rather than the CRM.
  3. The assertion. What would count as a pass.

The assertion is the hard part. Agents rarely produce the same words twice, so exact-match checks fail on harmless rewording, and people learn to ignore red builds. Assert on the outcome instead. The τ-bench benchmark from Yao and colleagues does this for tool-using agents: it compares the database state at the end of a conversation with an annotated goal state. That is the right model. Did the right thing happen to the world?

Useful assertions, roughly in order of how much you should trust them:

  • State checks. What changed in the mocked systems, and what did not.
  • Action checks. Which tools were called, with which arguments, and which were never called. "No call to send_email with an address outside the customer record" is a precise, cheap assertion.
  • Hard content rules. No credentials in the output, no account numbers in a reply to an unverified caller.
  • Judged checks. A judge model with a rubric, for properties code cannot check: did the agent disclose the required risk warning, was the refusal correct?

Building a golden set from real failures

A golden set is a collection of cases whose expected outcome a person has verified. It should grow from four sources:

  • Production failures. Every incident, complaint or flagged trace becomes a case once someone has confirmed what should have happened.
  • Red-team findings, written as runnable cases rather than prose.
  • Behaviour you must keep. The core tasks the agent exists for, and correct refusals: an agent declining something it should decline is a pass that a naive check can score as a failure.
  • Near misses. Cases that passed narrowly, or only on some runs.

Version the golden set alongside the agent's code and configuration. Never delete a case because it has been passing for months; that is what a regression case is supposed to do.

Which changes should trigger the agent test suite

Run the suite on changes, not on a calendar. An agent has more moving parts than its code:

ChangeWhy it mattersWhat to run
Model or model versionBehaviour shifts everywhere at onceFull suite
System promptSmall rewordings move edge casesFull suite
Tool added, removed or redescribedThe model reads tool descriptions as instructionsFull suite, plus the affected flows
Retrieval source or indexChanges what the agent believesCategories that use it
Judge, rubric or thresholdChanges what counts as a passRe-score the golden set against the last baseline
Framework or dependency upgradeCan change how tools are calledSmoke set

Tool descriptions deserve the attention. They are text the model trusts, and a third-party server can change one after you approved it, the rug pull described in MCP tool poisoning. Hash the descriptions in your test fixtures, and treat a changed hash as a change that triggers the suite.

How to handle non-determinism in LLM agent tests

Setting temperature to 0 does not make an agent repeatable.

  • Ouyang and colleagues' study of ChatGPT in code generation found that "setting the temperature to 0 does not guarantee determinism".
  • Horace He and colleagues at Thinking Machines Lab, in Defeating Nondeterminism in LLM Inference, note that even at temperature 0, "LLM APIs are still not deterministic in practice", and trace the main cause to batch size varying with server load.
  • OpenAI's own guide to the seed parameter describes it as best effort and says plainly: "Determinism is not guaranteed."

So reduce the variance you control, and measure the rest:

  • Fix what you can. Temperature 0, a seed where supported, a pinned model version, mocked tools, a fixed clock and fixed IDs. Record the full configuration with every run.
  • Run each case several times. One pass proves the agent can do it. τ-bench distinguishes pass@k, the chance that at least one of k trials succeeds, from pass^k, the chance that all k succeed. A regression gate cares about the second, because production gives the agent one attempt each time, many times a day. The τ-bench authors found the agents they tested "quite inconsistent (pass^8 <25% in retail)".
  • Don't retry until green. A case that passes on some runs and fails on others is not flaky test infrastructure. It is telling you the agent is unreliable on that input.

How to set a pass-rate gate per release

A single overall pass rate hides the failures that matter. Gate by category instead.

  • Safety-critical cases (unauthorised actions, data leaving where it shouldn't, correct refusals): every run of every case must pass.
  • Task-quality cases: a pass-rate threshold per category, agreed in advance with the people who own the agent.
  • Against the baseline: block when a case that passed reliably on the last release now fails on any run. Suppose each case runs five times. A case that went from five out of five to three out of five is a regression, even if the overall rate barely moved.

Two rules keep the gate honest. First, a run that crashed, timed out or came back with no verdict is a failure, not a skip; a missing verdict is never a pass. Second, pin the judge. If a model grades your judged checks, its version is part of the test, and a judge that varies between runs adds noise to every gate, one reason we argue the judge should be small and reproducible.

A CI checklist for agent regression tests

  1. Does every known failure have a test case with a verified expected outcome?
  2. Do assertions check outcomes and actions, not exact wording?
  3. Does every model, prompt, tool or configuration change trigger the suite?
  4. Does each case run more than once, with results reported per case?
  5. Are safety-critical categories gated at every run passing?
  6. Are crashed and unscored runs counted as failures?
  7. Is the judge's version pinned and recorded with each result?

Where Syntrox fits

Nucleus, our judge, runs at temperature 0 with a fixed seed, so the same input gives the same verdict, and every verdict is stamped with the judge model and version that produced it. A changed result can then be traced to the agent rather than to the grader. Our evaluation pipeline records any trace no judge could score as a failure, never as a pass. We hold the judge to the same discipline: its injection-resistance and evidence-order probes run as a release gate with explicit thresholds.