A single agent run can contain intent classification, retrieval summarisation, tool selection and a consequential final decision. These steps do not need equal reasoning capacity. Sending all of them to the strongest model wastes money on routine work; sending all to the cheapest model can damage the whole trajectory. Route each step instead, but optimise the completed task rather than a collection of isolated answers.
Two mechanisms offer different evidence for that decision. A pre-generation router predicts difficulty from the request and workflow state, then invokes one model. A confidence-driven cascade lets a cheap model answer, evaluates that response and escalates when necessary. The first avoids repeated generation. The second sees an actual answer before committing to a stronger model. Neither guarantees a correct route.
Choose before generation or after evidence
RouteLLM frames pre-generation selection as a learnt choice between a stronger, costlier model and a weaker, cheaper one. Its win-prediction model estimates the probability that the strong model will produce the preferred answer for a query. A threshold converts that probability into a route: raise the threshold to permit fewer strong-model calls. Preference comparisons supply training labels, while in-domain examples can align the router with the deployment distribution.
This design adds router computation to every step but only one generation. Its quality-cost operating point is explicit: plot response quality against strong-model use as the threshold changes. RouteLLM’s performance gap recovered and call-performance threshold formalise that comparison. They are more informative than a router accuracy score detached from the resulting answers.
An agent makes the input harder to define. The relevant request is not just the user’s opening message. It includes the step’s instructions, retrieved evidence, tool observations and compressed history. A router trained on general chat preferences may not recognise that a short tool result still requires a delicate decision. RouteLLM’s weak out-of-domain results without augmentation are therefore an operational warning, not a footnote. Train and test on the states the agent actually reaches.
FrugalGPT supplies the contrasting mechanism. Its cascade calls models sequentially and scores each query-answer pair for reliability. A score above the stage threshold accepts the answer; a lower score invokes the next model. The cheap answer has already been generated, so escalation pays for it as well as the stronger response. In return, the gate can use response-level evidence that a pre-generation router cannot see.
That evidence needs careful construction. FrugalGPT’s learnt reliability score is not a correctness proof, and a model’s self-reported confidence is not automatically calibrated. Prefer deterministic checks where the step permits them: schema validation, executable tests, required fields or verification against tool state. Add a calibrated quality estimator for open-ended judgements. Measure false accepts separately from unnecessary escalations. The former can release a bad action; the latter mainly spends time and money.
Replay representative traces, not benchmark prompts
Build the evaluation set from representative agent runs. Preserve ordinary requests, difficult reasoning, long contexts, ambiguous instructions, tool errors and recovery paths. Include rare steps whose mistakes carry serious consequences, even if they barely affect an average. Freeze the inputs, retrieved material and tool fixtures needed for replay, and record model, prompt and router versions.
Compare all-strong and all-cheap baselines with the candidate pre-router and cascade. Keep the candidate model pool, prompts, tools and step budgets fixed. During offline data collection, run the candidates needed to label quality and train the router. During policy evaluation, charge only the calls that policy would make. A cascade does not get free access to answers that its production gate would never request.
Score each step where a reliable oracle exists, then inspect the final environment state. Did the agent satisfy the user’s constraints, use admissible tools and avoid unauthorised side effects? End-to-end success must dominate: a cheap planner can choose a valid-looking action that makes later recovery impossible, while an expensive final writer cannot repair missing evidence. AgentBoard’s trajectory and subgoal analysis supports this process view, although its annotated progress measures do not remove the need for task-specific checks.
Do not demand one golden action sequence when several paths achieve the same state. Instead, retain per-step route decisions and outcomes to locate where trajectories diverge. Report quality by task slice and by route, including false cheap-route accepts and escalation rates. These diagnoses explain an aggregate result; they should not replace final success.
Account for the run’s cost and waiting time
Strong-model call rate is a useful proxy, not a monetary result. Calculate incurred cost from the prices and token counts applicable at evaluation time, including input, output, router and scorer calls. Historical paper prices describe historical experiments. They say nothing about a deployment’s current bill. Repeat the calculation when a provider, model or price changes.
Sum costs across the whole run, including abandoned cheap generations and any retries. Then report cost per successful task, not only cost per request. A policy that lowers spend per step but causes more failed runs may lose the comparison. Keep quality constraints visible by plotting the attainable task-success and cost frontier as routing thresholds vary, rather than presenting one favoured operating point as universal.
Latency needs the same whole-run treatment. Record time to first token where users stream output, but also total completion time at the median, p95 and p99. A pre-router adds a decision stage to every applicable step. When the cheap answer passes, a cascade returns after one generation; its escalated cases perform generation, scoring and regeneration serially. Those cases can dominate the tail even when the mean looks attractive. Several routed steps accumulate these delays, so measure whole-run distributions rather than adding step-level percentiles.
Set step and run deadlines before choosing thresholds. A tight latency objective may favour direct routing or immediate strong-model selection for steps likely to escalate. A loose batch workflow may tolerate a longer cascade for lower cost. Cap escalation depth, and do not hide speculative parallel calls in the latency figures: they change the cost policy as well.
Put escalation and failure on separate paths
A practical policy can combine the mechanisms. Start with deterministic eligibility rules for required modalities, tool support, context limits and data restrictions. Use a learnt pre-router where query features reliably separate routine from difficult steps. Add a response gate where acceptance can be checked with useful evidence, especially before a consequential action. If neither signal is dependable, the strong baseline may be simpler and cheaper than maintaining a router.
Low confidence is not an operational failure. A valid but doubtful answer triggers quality escalation. A timeout, rate limit, malformed transport response or provider outage triggers reliability fallback: bounded retry, an approved alternative route, a queued attempt or a safe stop with human handoff. Keep these events and their metrics distinct. Otherwise an availability retry can be mistaken for evidence that the first answer was poor, and an escalation loop can repeat a failing call without a terminal state.
Test every fallback independently under injected faults. An alternative model must satisfy the same capability and data controls, and exhaustion must leave the agent in a safe state. More model choices do not themselves provide resilience.
Log the step identifier, route features, policy version, score, threshold, selected model, escalation reason, token use and latency. Sample accepted cheap answers for review; escalation logs alone cannot reveal false accepts. Recalibrate after model, prompt, tool or workload changes, and re-run frozen traces before rollout. Routing earns its place when it preserves end-to-end success at a better measured cost-latency point—not when it merely reduces strong-model calls.
Sources and further reading
- Ong et al., “RouteLLM: Learning to Route LLMs with Preference Data”, v3 (21 July 2024) — preference-trained pre-generation routing, cost thresholds, quality-cost metrics, router overhead and out-of-domain limitations.
- Chen, Zaharia and Zou, “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance”, v1 (9 May 2023) — sequential model cascades, learnt generation-reliability scores, acceptance thresholds and budget-constrained optimisation.
- Moslem and Kelleher, “Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey”, v2 (21 April 2026) — a taxonomy of decision timing, routing signals, uncertainty, compositional systems and evaluation measures.
- OpenAI, “A practical guide to building agents” (15 January 2026) — per-task model substitution guided by accuracy evaluations, complexity, cost and latency.
- Zhang et al., “AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents”, v2 (23 December 2024) — multi-turn trajectories, final success, subgoal progress and the limits of annotated agent evaluation.
- MLflow, “LLM Routing in Production: Four Stages, Mapped to MLflow” (7 September 2026) — contemporary production guidance on per-step routing, cascade latency, trace telemetry and provider-failure circuit breakers.