LLM routing needs evaluations, not intuition
A router is a policy that makes a model decision for every request. The right way to improve it is to evaluate the whole system—including fallbacks and cost—not just the underlying models.
LLM routing sends each request to a model according to a policy: capability, task type, cost, latency, availability, or learned preference. Evaluate the router as a system. Measure accepted-answer quality, total cost, latency, failures, fallback behavior, and the stability of decisions as models and prices change.
Why route instead of choosing one model?
One model rarely dominates every workload. A fast inexpensive model may handle classification and extraction well, while a larger reasoning model earns its cost on ambiguous analysis. Some requests require vision, long context, a specific region, or tool calling. Routing turns those differences into a per-request decision instead of forcing every task through the most expensive option.
Research supports the basic idea but not a universal savings promise. RouteLLM trained routers on human-preference data to choose between a stronger and weaker model; its paper reported more than twofold cost reductions in some benchmark settings with little quality loss. FrugalGPT studied a related cascade approach that conditionally calls models. Those results belong to the tested models, prompts, thresholds, and datasets—not to every production router.[1][3]
A router is a policy, not a magic classifier
The policy may be a table of rules, a semantic classifier, a small language model, a cascade, or a learned preference model. It maps signals—requested alias, modality, prompt features, tenant controls, price, current health—to an eligible destination. A production router also needs a refusal path when no allowed model satisfies the request.
Eligibility should come before optimization. Security boundaries, geography, data policy, required context, tool support, and customer configuration narrow the candidate set. Only then should the policy optimize quality, cost, or latency. Otherwise a cheaper choice can violate a requirement the scoring function never represented.
Evaluate decisions on a routing-shaped dataset
Build a dataset that represents the traffic mix: short factual questions, extraction, code, long-context synthesis, multimodal inputs, adversarial prompts, and invalid requests. Label the required capabilities and, where possible, score candidate responses against a task rubric. Preserve segments because an average can hide that the router saves money by degrading one small but important class.
The offline replay should compare at least three baselines: always use the strongest eligible model, always use the cheapest eligible model, and the proposed router. Report quality, cost, and latency together. If the router cannot beat a simple rule on the decision surface you care about, the extra complexity is not justified. OpenAI’s eval framework similarly treats model and system changes as hypotheses to test against use-case data, while NIST stresses that benchmark suitability depends on the intended use of the result.[2][4]
| Metric | What it reveals | Common trap |
|---|---|---|
| Accepted-task rate | End-to-end usable outputs | Using a generic judge without calibration |
| Cost per accepted task | Economic efficiency | Counting cheap failed answers as savings |
| p95 latency | Tail experience | Reporting only an average |
| Fallback success | Resilience under failure | Ignoring the cost and model actually used |
Choose thresholds from a frontier, not a slogan
A learned router often produces a score that estimates whether a stronger model will help. The threshold controls how often to escalate. Sweep that threshold and plot quality against cost. The resulting curve shows the achievable frontier for the tested traffic. Product owners can then choose a point based on the cost of errors rather than declaring that one threshold is objectively best.
Use different thresholds for different risk classes when justified. Drafting a headline and interpreting a contract clause do not carry the same downside. But keep the policy legible: too many untested exceptions turn the router into an accidental collection of production folklore.
Test fallbacks as deliberate behavior
Fallbacks handle provider errors, rate limits, timeouts, and unavailable models. They can preserve availability, but they also change output style, tool syntax, latency, and price. A fallback should remain within the original eligibility boundary and record which model ultimately served the request.
Chaos tests are useful here: disable the preferred provider, inject timeouts, remove a capability, and change a rate card. Verify that the system either selects an allowed alternative or fails clearly. A silent fallback to an ineligible model is worse than a visible error because it makes compliance and debugging depend on guesswork.
Leave a routing receipt
Every response should expose enough information to explain the decision without revealing provider secrets. A receipt can include an opaque receipt ID, public model alias, policy and catalog versions, selection reason, confidence, fallback chain, settled token usage, and charged units. That lets support teams connect a user-visible outcome to the policy that produced it.
Versioning matters because a model alias, price, or route can change. Without the policy and catalog version, two identical prompts sent a week apart may be impossible to compare. Receipts turn routing from hidden infrastructure into an auditable product surface.
Monitor drift and rerun the suite
Production traffic changes, providers update models, and prices move. Track routing share by segment, fallback rates, cost per accepted task, latency, user corrections, and safety incidents. Compare these against the offline expectation. A sudden shift may indicate task drift, a health problem, or a router that has learned an unintended shortcut.
Gate policy releases through shadow traffic or controlled experiments. Store the candidate decision without changing the live route, then compare what would have happened. When confidence is sufficient, ramp gradually and retain the old policy version for rollback.
Frequently asked questions
What is LLM routing?
LLM routing is the selection of an eligible model or provider for each request according to a policy based on capabilities, task features, cost, latency, health, or learned preferences.
How do you measure whether an LLM router is good?
Compare end-to-end quality, cost per accepted task, latency, failures, and fallback behavior against strong and cheap baselines on representative traffic.
Should users see which model was selected?
A routing receipt should expose a public alias and decision metadata sufficient for explanation and billing, while keeping internal provider routes and sensitive configuration private.
Sources and evidence
Primary and authoritative sources used for factual claims. Company research and executive forecasts are labeled as such in the article.
- 1RouteLLM: Learning to Route LLMs with Preference DataUC Berkeley, Anyscale, LMSYS · 2024-06-26
- 2OpenAI EvalsOpenAI
- 3FrugalGPTStanford University · 2023-05-08
- 4