A confident final answer can conceal a mistaken lookup, an ignored error and an unauthorised write. Even a correct final state offers incomplete evidence: the agent may have damaged unrelated records before reaching it. The opposite blind spot is just as costly. Comparing a run with one reference action sequence punishes valid detours and alternative methods.
Evaluate the run as a sequence of decisions in a changing environment. Final-state checks establish whether the task was completed. Action and observation logs explain how it behaved. Subgoal checks expose partial progress, controlled faults test recovery, and invariants catch collateral damage. These signals answer different questions; none can replace the others.
Turn the run into checkable evidence
Model an episode as an initial state followed by actions, observations and resulting states. The distinction matters. A screenshot, tool response or accessibility tree is a partial observation. The authoritative state might include database rows, files, application settings and approval records that the agent cannot see directly. Evaluation should inspect that state rather than trust the agent’s description of it.
Capture enough structure to reconstruct each transition: the proposed action and arguments, execution status, returned observation, error, state version and timestamp. Add policy decisions such as approval requests and confirmations when they constrain an action. Keep model, prompt, tool and environment versions with the run. Do not turn a generated rationale into ground truth; evaluate the observable decision and its effect.
Start from a controlled initial state. WebArena resets functional web applications and checks task outcomes programmatically, allowing different routes to satisfy one intent. OSWorld goes further by configuring real computer environments, executing keyboard and mouse actions, then retrieving files or application state for task-specific evaluation functions. Its design explicitly rejects non-execution comparisons that assume one solution.
This machinery does not make every check objective by default. A locator can inspect the wrong field, and an evaluator can encode an unnecessarily narrow interpretation. Test the harness with several acceptable solutions and with near misses that must fail. Treat checker defects as evaluation defects, not as agent failures.
Measure progress without prescribing a script
Binary success hides the difference between an agent that never found the relevant record and one that completed every step except confirmation. AgentBoard addresses this gap with a progress rate based on the best goal-state or annotated-subgoal match reached during a run. The useful mechanism is a state-dependent view of progress, not a more generous pass mark.
Define subgoals as predicates over evidence: the customer was identified, the requested order was located, the price was confirmed and the refund was recorded. Express only genuine ordering constraints. Locating a customer and reading a policy may be independent; charging a refund must follow confirmation. A partial order or graph of required conditions permits alternative tool calls, searches and harmless detours while preserving the steps that make completion valid.
ToolSandbox offers a related design. Its milestones describe events that must occur, with temporal dependencies, while its evaluator finds the best chronological match across a trajectory. Its minefields describe events that must not occur. Both need domain knowledge and careful authoring; the paper identifies that effort as a scaling limitation. They are task-specific evaluation structures, not a universal trajectory language.
Keep two views of progress. A high-water mark shows how far the agent reached, which helps diagnose an abandoned or stalled run. Current and final-state checks show whether it retained that progress. AgentBoard’s best-match measure can otherwise hide a later regression: locating the right order earns progress even if a subsequent action modifies the wrong one.
Test recovery with controlled faults
A successful clean run cannot demonstrate recovery. Introduce one controlled failure at a chosen transition: a timeout, stale element, malformed tool response, rejected argument, unexpected window or corrected user detail. Preserve the same initial state and compare the disturbed run with an undisturbed control.
Recovery is a sequence, not a retry count. The agent should incorporate the error observation, stop repeating an invalid action, establish any missing precondition and then resume the task. In ToolSandbox’s state-dependency example, sending a message fails while cellular service is off. Enabling service and retrying demonstrates a causal repair; repeatedly issuing the same send call does not. A later success still needs a duplicate-message check.
Report detection and recovery separately. Useful measures include repeated invalid actions, steps to a valid transition, recovery rate by fault type and additional latency or tool calls. Also test whether the agent stops safely when recovery exhausts its budget. Do not infer these behaviours from a fluent account of what the agent intended to do.
Fault injection needs restraint. Choose faults that production interfaces can actually return, keep their payloads consistent with the tool contract, and make the injected point visible in the evaluation record. An arbitrary corrupted response may create an impossible puzzle rather than evidence about resilience.
Put harmful side effects in the pass condition
Write the goal as required changes plus protected state that must remain unchanged. For a refund, verify the target order, amount and payment destination, then compare unrelated orders, customer settings and outbound messages with their initial snapshots. For a file edit, inspect the intended artefact and protected files rather than accepting a screenshot of the final window.
Minefield-style checks can reject a forbidden call even when the goal predicate passes. Event checks also catch effects that a final snapshot misses, such as sending data to an external service and then deleting the local evidence. Use state comparisons for unauthorised mutations and action-log assertions for prohibited transmissions or missing approvals.
A correct outcome alone cannot establish policy compliance. τ-bench compares the final database with an annotated goal state and permits varied conversations and read actions, yet its authors note that this reward can miss a missing confirmation. The trajectory must therefore supply separate predicates for consent, ordering and other process rules.
Software evaluation offers the same distinction. SWE-bench checks fail-to-pass tests for the requested fix and pass-to-pass tests for preserved behaviour. Agent evaluation needs both ideas at episode scope: achieve the new goal without breaking existing guarantees. A serious invariant violation should fail the run, not merely reduce an aggregate quality score.
Build a scorecard, not a universal judge
Keep mandatory gates separate from diagnostics. Goal-state predicates and safety invariants determine acceptance. Required process predicates enforce domain rules. Subgoal curves, invalid-action counts, recovery and cost explain behaviour and guide engineering work. Combining them into one weighted number lets efficiency or partial progress compensate for damage that should block release.
There is no universal automatic judge hidden in these techniques. AgentBoard relies on human-annotated subgoals and sometimes simplifies tasks to obtain a unique subgoal sequence. OSWorld depends on custom getters and evaluators. ToolSandbox requires carefully authored milestones and minefields. Each exchanges broad coverage for precise, task-local evidence.
Language-model judging still has a narrow role where an answer’s wording varies. WebArena uses exact and substring checks for some answers and model-based semantic matching for others. Apply such matching to a specified answer property, calibrate it on reviewed examples and retain deterministic checks for state and side effects. Do not ask a model to infer the whole trajectory’s safety from a polished transcript.
Repeat stochastic episodes from identical initial states. τ-bench’s distinction between average success and success across every repeated trial matters operationally: one lucky completion does not establish dependable behaviour. Retain per-run traces and outcomes so variability remains visible.
Carry the evidence into release decisions
Compare candidate and incumbent systems on the same versioned tasks, policies, fixtures and fault injections. Report results by task type, fault and invariant severity, with cost and latency beside them. Include realistic end-to-end environments as well as controlled fixtures; a mock can omit the race condition or partial write that decides safety.
Review ordinary passes as well as flagged failures. Alternative valid solutions expose overfitted checkers, while sampled production traces reveal conditions absent from the suite. Limit exported logs to the fields needed for evaluation and govern their access and retention like any other sensitive operational data.
The practical unit of evaluation is not an answer or a golden path. It is a checked transition history ending in an acceptable world state, with evidence that the agent can recover without causing damage elsewhere.
Sources and further reading
- Zhang et al., “AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents”, v2 (23 December 2024) — action–state trajectories, best-match subgoal progress, annotation limits and support for varied detours.
- Xie et al., “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments”, v2 (30 May 2024) — controlled computer setup, execution-based final-state evaluation and the case against one assumed solution.
- Lu et al., “ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities”, v1 (8 August 2024) — mutable world state, milestone graphs, minefields, error-driven recovery and authoring limitations.
- Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”, v1 (17 June 2024) — final-database comparisons, variable interaction paths, repeated-trial reliability and the limits of outcome-only rewards.
- Zhou et al., “WebArena: A Realistic Web Environment for Building Autonomous Agents”, v4 (16 April 2024) — functional correctness, programmatic state checks and multiple valid execution paths.
- Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, v2 (5 April 2024) — executable evaluation with fail-to-pass requirements and pass-to-pass preservation checks.