A receipt audit can produce the right decision from the wrong extraction. The final answer passes by accident, while the trace reveals faulty intermediate reasoning. An end-to-end score alone can neither expose that accidental success nor localise a different failure.
Tracing supplies the missing execution record. Evaluation supplies a judgement about that record. Teams need both, joined by domain review, to convert an incident into a test that constrains the next prompt, model or tool change. The purpose is to preserve a specific failure, state what should have happened and make that expectation repeatable, not to archive every conversation or appoint another model as arbiter.
Start with the trace, not a verdict
A useful trace links the steps of one run: the request, retrieved context, model calls, tool selection and arguments, tool results, retries and final action. Timestamps, token use and errors explain operational behaviour. Prompt, model, application and tool versions make the run comparable with another. Parent-child relationships matter when parallel subagents contribute to the same outcome.
This detail supports diagnosis by observable stage. Did retrieval omit the governing document? Did the model choose an unsuitable tool, or choose correctly but malformed its arguments? Did the tool fail, did the agent ignore its error, or did the final response misrepresent a successful result? These categories are working labels, not a universal taxonomy. Their value lies in assigning the next investigation to a concrete interface.
A trace is not ground truth. It records what the system emitted and observed, not whether those contents were correct or complete. An absent event may reflect missing instrumentation rather than an action that never happened. A model’s explanation may rationalise a decision without revealing its cause. Even a successful database write does not establish that the write was authorised or desirable.
Domain feedback closes that gap. A reviewer should inspect the evidence available at the suspected step, then record the expected outcome, unacceptable behaviour and severity. Sometimes the right correction is a policy clarification rather than a model change. Reviewer disagreement can expose an ambiguous requirement; forcing one label would conceal it.
Record the review target as precisely as the failure. A final answer may be acceptable despite a wasteful detour; an efficient trajectory may still violate a policy. Separate labels for task success, process compliance and evidence quality prevent one pass/fail bit from collapsing these concerns. Where reviewers supply a corrected answer, preserve its supporting source or rule. Otherwise the eval inherits an unexplained preference that cannot be audited when policy changes.
Severity should shape the test’s release consequence. A formatting defect may warrant a trend, while an unauthorised write must block deployment. Frequency answers a different question: a rare destructive action deserves a regression case even when it barely changes the aggregate failure rate. This is also why incident counts alone cannot prioritise all evaluation work.
Turn the failure into a controlled case
Preserve the smallest case that reproduces the problem. Store a permitted input or a redacted surrogate, the necessary context and tool fixtures, the reviewed expectation and the original failure stage. Add provenance so a future reviewer can distinguish a production incident from a synthetic variation. Record which system and trace versions supplied it.
Replay needs control over the environment. A live web search or mutable customer database can make an old failure impossible to interpret. Snapshot retrieved documents, simulate tool responses and direct side effects to a sandbox where possible. Keep at least some end-to-end cases against realistic services, because mocks can omit timeouts, partial responses and other integration faults.
Do not freeze the original trajectory as the only acceptable solution. Agents can reach the same outcome through different reasonable sequences. Assert the properties that matter: permitted tools, valid arguments, required evidence, final state and answer quality. Add a trajectory expectation only when the process itself is constrained, such as obtaining approval before a payment.
Keep the regression set separate from a representative workload sample. The former concentrates on known failures; the latter estimates behaviour across routine tasks and important slices. A candidate can fix every old incident while damaging ordinary work. Both sets need versioning, and cases used to tune a prompt should not be the sole evidence of improvement.
Layer the offline evaluation
Begin with deterministic checks wherever the requirement permits them. Validate output schemas and data types. Compare extracted fields with labels, recalculate totals, inspect final state, reject forbidden tool calls and verify that citations resolve to supplied sources. These oracles are repeatable and cheap, but their scope is narrow: valid JSON can still contain a fabricated account number.
Next, evaluate intermediate components with controlled inputs. Give the decision stage the correct retrieved facts to test its reasoning separately from retrieval. Then run the complete workflow on the original input. Both views matter. A faulty extraction can coexist with a correct final decision, while a sound decision stage cannot repair evidence it never received. Component scores localise work; end-to-end scores protect the user-visible result.
Open-ended quality needs a different oracle. An LLM judge can apply a rubric for factual support, completeness or source quality, but its score is another model output. Calibrate it against domain-labelled examples from the same task, including borderline cases. Measure false accepts and false rejects by slice rather than reporting one agreement average. Check stability under irrelevant lengthening and, for pairwise grading, swapped answer order. The MT-Bench study documented position and verbosity biases alongside strong human agreement in its own setting; neither result transfers automatically to a new rubric.
Pin the judge model and prompt with the eval suite, and recalibrate when either changes. Give the judge trusted references where available rather than asking it to recall specialised facts. Keep its rationale as debugging evidence, not proof. A judge that shares the candidate’s blind spots can make two wrong systems agree.
Human review remains the discovery mechanism and the court of appeal. Route judge disagreement, severe errors and novel patterns to reviewers, but also sample ordinary passes. Reviewing only flagged cases cannot reveal failures that every automated check missed. Experts should correct outputs and refine rubrics; their time is better spent resolving uncertainty than repeating an exact assertion that code already handles.
Gate versions, then watch reality
Run candidate and incumbent on the same frozen cases with comparable tools and settings. Repeat runs when generation varies, and retain per-case outcomes. Gate on critical assertions and protected slices, not only a change in average score. Include task cost, latency and escalation rates so an apparent quality gain does not depend on an unacceptable number of retries or reviewer hours. Small suites can expose large faults, but they cannot support precise claims about modest improvements.
Passing the gate does not guarantee production behaviour. Monitor sampled traces, user corrections, tool errors and outcome rates for drift and unfamiliar failures. Aggregate decision patterns can reveal a shift without requiring staff to read every conversation. Monitoring discovers conditions; repeatable offline evals compare system versions. Confusing those roles produces either unreviewed anecdotes or a suite that never learns.
Privacy must shape the instrumentation itself. Prompts, retrieved documents and tool arguments can contain personal or commercially sensitive data, and copying them into a longer-lived eval set creates another store to govern. Collect only necessary fields, redact before export, restrict reviewer access and set deletion periods. Keep correlation identifiers separate from identity, and never place sensitive values in trace-propagation headers. Where content is unnecessary, record structural features such as tool sequence, status and duration instead.
The durable asset is the reviewed connection between an observed run and an expected behaviour. Each incident can then improve the suite without turning logs into labels, judges into authorities or monitoring into surveillance.
Sources and further reading
- OpenAI, “New tools for building agents” (11 March 2025) — the Agents SDK release and integrated tracing for inspecting agent workflow execution.
- OpenAI Cookbook, “Eval Driven System Design: From Prototype to Production” (2 June 2025) — domain-labelled examples, component and end-to-end graders, business trade-offs and production feedback.
- Anthropic, “How we built our multi-agent research system” (13 June 2025) — flexible agent trajectories, rubric-based judging, human evaluation, production tracing and privacy-conscious monitoring.
- Yehudai et al., “Survey on Evaluation of LLM-based Agents”, v1 (20 March 2025) — capability, application, trajectory and fine-grained evaluation, with cost and safety limitations.
- Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, v4 (24 December 2023) — human agreement, judge calibration methods and position, verbosity and reasoning limitations.
- W3C, “Trace Context”, W3C Recommendation (23 November 2021) — trace correlation, privacy considerations and the prohibition on sensitive data in propagation fields.