A model benchmark can help shortlist a model. It cannot decide whether a particular application is safe to release. That decision depends on the prompt, retrieved context, tools, output handling and users surrounding the model. A gain on a general leaderboard may coincide with broken JSON, slower answers or a regression on one important customer workflow.
Production evaluation therefore needs a different unit of analysis: the whole application, exercised on tasks it actually performs. The useful artefact is not a single score but a versioned suite whose results can block or admit a release.
Build the suite from work and failure
Start with a sample of real requests, not a collection of conspicuously difficult prompts. Preserve the distribution of routine work, then add strata for long inputs, unusual formats, sensitive domains and other conditions that matter to the service. Production traces can supply both inputs and surrounding context, provided the team has permission to use them and removes data that evaluation does not need.
Each case should contain more than an input. Record the expected behaviour, an acceptable answer or grading rubric where possible, and the severity of a failure. Add provenance and slice labels so a result can be traced back to a workflow rather than disappearing into an average. A retrieval case may need the expected document identifiers and frozen source text; an extraction case may need a schema and field-level labels.
Observed failures deserve a separate regression set. When a user finds a missing citation, an invalid tool call or an unsupported refusal, minimise the example and add it with a precise expectation. Keep the broader representative sample intact. If every new failure enters that sample and subsequent prompts are tuned against it, the suite gradually measures familiarity with old bugs rather than performance on the current workload.
Version the cases, rubrics and expected answers alongside the application configuration. A suite revision is itself a change: adding harder cases can lower a score without making the application worse. Reports should identify both the system version and the suite version.
Use the cheapest reliable oracle
Different outputs permit different checks. Use code wherever code can express the requirement: parse structured output, validate its schema, compare an exact label, run a calculation, inspect required source identifiers or reject a forbidden action. These checks are cheap to repeat and their failures are easy to localise.
Behavioural testing supplies a useful structure here. CheckList distinguished minimum-functionality cases, invariance under changes that should not affect the answer, and directional tests where a change should affect it in a known way. An LLM application can adapt all three. A request must contain every required field; changing a customer’s name must not alter an extracted amount; removing a source passage must remove claims that depended on it.
Determinism is not the same as validity. Exact string matching can reject a correct paraphrase, while a schema validator will accept plausible nonsense if every field has the right type. Keep deterministic assertions narrow and explicit about what they do not test.
Open-ended answers need judgement. A model grader can apply a rubric at a scale that manual review cannot, but first calibrate it on examples that people have graded for the same application. Cover clear passes, clear failures and borderline cases. Measure disagreement by slice and inspect false accepts separately: a judge that overlooks unsupported claims is not interchangeable with one that marks sound answers too harshly.
Make the rubric concrete. Ask whether each required claim is supported, whether the answer follows a stated constraint and whether it omits necessary qualifications. Supply trusted references where they exist. For pairwise comparisons, hide system identities and swap answer positions; inconsistent verdicts should become ties or enter human review rather than choosing a winner arbitrarily. Pin the judge model and prompt because a judge upgrade can change every result without any change to the candidate.
The original MT-Bench study found strong agreement between GPT-4 and human preferences in its setting, exceeding 80%. It also exposed position and verbosity biases and weaker judgements on mathematics and reasoning. Neither the agreement nor the failure rates transfer automatically to a production rubric. Calibration is a local, repeatable measurement, not a property permanently possessed by a model.
Keep people in the decision loop
Human review should resolve the cases where the oracle is weakest. Send it judge disagreements, novel failure patterns and outputs whose errors carry serious consequences. Also review a random sample of apparently ordinary passes; inspecting only flagged outputs leaves the judge’s blind spots untouched.
Reviewers need the same task context and a rubric that separates factual support, instruction following and usefulness. Where judgement is subjective or risk is high, use independent labels and adjudicate disagreement. That disagreement may reveal an ambiguous requirement rather than an annotator error. Periodic relabelling also exposes reviewer drift.
People should improve the suite, not merely certify individual answers. Turn reviewed failures into reusable cases and revise vague rubrics. Resist converting every preference into a rigid rule: some differences reflect taste, and codifying them can make later evaluations reward house style instead of task success.
Gate the release on paired evidence
Run the candidate and incumbent against the same frozen suite under comparable conditions. Record the model revision, prompt, decoding settings, retrieval corpus, tool implementations and application code. Without that bundle, a stored score cannot explain what changed or reproduce a rollback decision.
Paired per-case comparisons show where the candidate differs. Report pass rates by important slice, not just the overall mean, and retain individual failures for inspection. Repeat sampled generations when output varies. A small aggregate improvement can conceal a severe regression in a small but essential workflow; uncertainty around a handful of cases should not be presented as a reliable win.
Quality is only one constraint. Calculate cost from actual prompt and completion tokens, retrieval and tool charges, retries and any extra model calls. Cost per successful task is often more informative than cost per request because a cheap answer that fails may trigger another attempt or human work. Evaluation itself also consumes tokens and reviewer time, so include the judge when comparing operating designs.
Measure latency through the application path, including retrieval, queueing and tool calls. Percentiles and timeout rates expose tail behaviour that an average hides. HELM’s efficiency analysis warned that hardware, serving software and contention confound naive cross-provider runtime comparisons. The practical response is not to discard latency, but to measure the systems available to you under controlled, representative load.
A release policy can then require every critical deterministic test to pass, no unacceptable regression in protected slices, stable judge–human agreement, and cost and tail latency within their budgets. A candidate that misses a quality floor does not become acceptable because it is cheaper. One that exceeds the latency budget does not pass because its prose improves. Keep these constraints visible instead of compressing them into a weighted score whose weights nobody can defend.
Treat the gate as a boundary, not a guarantee
An offline suite cannot contain future requests, every adversary or rare interactions between tools. External models and data sources also change. Continue sampling production outcomes, run a limited release where feasible and retain the previous application bundle for rollback. Re-run the incumbent beside the candidate when an unversioned upstream service may have drifted; otherwise old baseline results become a moving reference.
This process does not prove general intelligence or settle what GPT-4 can do in abstract. It answers an operational question with bounded evidence: does this application still perform its assigned work within explicit quality, cost and latency limits? A maintained suite makes that answer repeatable while leaving room for human judgement where measurement ends.
Sources and further reading
- Ribeiro, Wu, Guestrin and Singh, “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList” (2020) — capability-led minimum-functionality, invariance and directional tests.
- Liang et al., “Holistic Evaluation of Language Models” (HELM), v1 (2022) — scenario-specific, multi-metric evaluation and the difficulties of comparing inference efficiency.
- OpenAI, “GPT-4” research announcement, OpenAI Evals section (14 March 2023) — sample inspection, custom evaluation logic, model-version tracking and human evaluation alongside automated tests.
- OpenAI Evals documentation at the 26 September 2023 revision — the framework’s treatment of evals for models and complete LLM systems, including prompt chains and tool-using agents.
- Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, v1 (2023) — human-preference agreement, judge biases and reference-guided grading.
- Chen, Zaharia and Zou, “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance”, v1 (2023) — task performance under a cost budget and the components of LLM API cost.