How to Build an Evaluation Dataset for an AI Agent: Cases, Criteria, Labellers and Agreement
Build the golden set in seven steps: source cases from real traffic; write binary, observable pass/fail criteria before labelling; have at least two people label an overlap and measure chance-corrected agreement; map coverage with a task-by-condition matrix; size each category for the confidence interval you need; version cases, criteria and labels; and refresh the set while keeping part of it out of prompt tuning.
On this page
A golden set is the collection of cases whose correct outcome a person has verified, and against which every version of an agent is scored. Our guide to regression testing LLM agents in CI covers running one on every change. This post is about building one worth running: where the cases come from, who decides what passes, and how you know the labels mean anything.
Much of the error in an agent evaluation sits in neither the agent nor the grader. It sits in the dataset: cases that don't resemble production, criteria nobody wrote down, and labels one person assigned late on a Friday.
Step 1: Source cases from real traffic
Start from production, or the closest thing you have.
- A random sample of real traces, so the set reflects what the agent actually sees. Stratify by task and channel, so rare but important tasks are present.
- Every incident and complaint, once someone has confirmed what should have happened.
- Red-team findings, written as runnable cases rather than prose.
- Synthetic cases for the gaps your coverage review finds (Step 4), marked as synthetic, so you can report results with and without them.
Real traces contain real customers' data. Scrub or tokenise a trace before it becomes a test case, and give the golden set retention rules of its own, with the same discipline as keeping PII and secrets out of agent traces.
Step 2: How to write pass/fail criteria for an AI agent
"Was this a good response?" is not a criterion. Two careful people will answer it differently, and each will be right by their own standard.
Write criteria that are:
- Binary and separate. "Resolved the request", "followed the refund policy" and "gave the required warning" are three pass/fail checks, not one score out of ten.
- Observable in the trace. A labeller should be able to point at the line that makes the case pass or fail.
- Anchored with examples, above all a borderline pass and a borderline fail for each criterion.
Expect the criteria to move once people start labelling. Shankar and colleagues, in Who Validates the Validators?, call this criteria drift: "users need criteria to grade outputs, but grading outputs helps users define criteria." Plan for it. Run a pilot round on a few dozen cases, revise the criteria, and version them, so every label records which version it was made against.
Step 3: How to measure labeller agreement
A label from one person is an opinion. Have at least two people label an overlapping sample independently, without seeing each other's answers, and compare.
Raw percent agreement flatters you, because two labellers agree some of the time by chance, and very often when nearly every case passes. In 1960 Jacob Cohen criticised percent agreement for exactly this, and introduced Cohen's kappa: a chance-corrected statistic for two raters that runs from −1 to +1, where 0 is the agreement chance alone would produce. McHugh's review, Interrater reliability: the kappa statistic, is a clear introduction. It warns that Cohen's suggested interpretation may be too lenient for health research, because it implies a score as low as 0.41 might be acceptable. Decisions about customers' money deserve the stricter reading.
For more than two labellers, or when not everyone labels every case, use Krippendorff's alpha, which handles any number of observers and missing data. Krippendorff's own note on computing alpha works through it step by step.
Agree the target per criterion before you measure. Then treat disagreement as information:
- Many disagreements on one criterion mean the criterion is unclear. Rewrite it.
- A few disagreements usually mark genuinely hard cases. Adjudicate them with a third person, record both the final label and the fact that it was disputed, and keep them. Disputed cases are often the most useful in the set.
- Low agreement everywhere means no grader, human or model, can be measured against these labels. Fix that before you trust any score.
Step 4: How to check coverage of edge cases
Ribeiro and colleagues' CheckList organises behavioural tests as a matrix of capabilities against test types. The same idea works for agents: put the agent's tasks on one axis and these conditions on the other.
| Condition | What a case looks like |
|---|---|
| Happy path | A clear request with everything the agent needs |
| Ambiguous request | Two plausible readings; the right move is to ask |
| Missing or conflicting data | A required record absent, or two sources disagreeing |
| Must refuse | A request outside the agent's mandate |
| Adversarial | An instruction hidden in a document, email or tool result |
| Tool failure | A timeout, an error or an unexpected response |
| Hand-off | A case that should go to a person |
Count the cases in each cell. Empty cells are blind spots, and they are where synthetic cases earn their place.
Step 5: How many test cases each category needs
Small categories give wide error bars. NIST's Engineering Statistics Handbook recommends the Wilson method for a confidence interval on a proportion. Suppose the agent passes all 30 cases in a category. The 95% Wilson interval for its pass rate runs from about 89% to 100%, so a true failure rate as high as 11% is consistent with a perfect score. With 300 cases and no failures, the upper bound on the failure rate falls to about 1.3%.
Two further corrections come from Evan Miller's Adding Error Bars to Evals. When cases come in related groups, such as several turns from one conversation or several requests from one customer, compute clustered standard errors; treating them as independent overstates your precision. And when comparing two versions of an agent, analyse the case-by-case paired differences rather than two overall scores.
Step 6: How to version an evaluation dataset
Every case needs an ID, its source, the date it was added, the criteria version, who labelled it, and whether it was disputed. Every result should record the dataset version it was scored against.
Gebru and colleagues' Datasheets for Datasets proposes documenting a dataset's motivation, composition, collection process and recommended uses. A one-page datasheet for the golden set answers the questions an auditor, or a new team member, will ask first.
When criteria change, relabel the cases they affect and say so in the changelog. A score against version 7 and a score against version 8 are different measurements.
Step 7: Keep it fresh, and keep part of it out of the prompt
An agent's traffic changes: new products, new policies, new ways customers phrase things. Sample fresh production traces on a schedule, label them, and compare their pass rate with the golden set's. A widening gap means the set no longer represents production.
Guard against contamination as well. LiveBench was built because test data that ends up in a model's training set "can quickly render benchmarks obsolete". Inside an agent team the leak is quieter: golden cases copied into few-shot examples, or a prompt tuned against the same cases until it has effectively memorised them. Hold back a portion of the set that is never used while tuning, and report it separately.
A golden set checklist
- Are cases drawn from real traffic, with synthetic cases marked?
- Are criteria binary, observable, anchored with examples and versioned?
- Do at least two people label an overlapping sample, with chance-corrected agreement reported per criterion?
- Are disputed cases adjudicated and kept?
- Does every cell of the coverage matrix have cases?
- Is a confidence interval reported for each category?
- Is a held-out portion kept away from prompt tuning?
Where Syntrox fits
A golden set measures the grader as well as the agent. Nucleus, our judge, runs at temperature 0 with a fixed seed, so the same case gets the same verdict, and every verdict is stamped with the judge model and version that produced it. It returns uncertain rather than guessing when a grounded check has no context to check against, which gives you a label to count rather than a pass to doubt.