LLM cost, quality, and latency: choose the efficient frontier

One winner rarely exists across all three axes

A large reasoning model may improve difficult analysis while increasing price and wait time. A small model may be ideal for extraction but fail on ambiguous instructions. A fast provider can still create poor tail latency during rate limits. Selection is therefore a constrained optimization problem, not a ranking contest.

The market also moves quickly. Stanford’s 2025 AI Index documented substantial declines in inference cost for systems reaching a fixed level of benchmark performance. That trend makes old model comparisons expire quickly and reinforces the need to preserve the date, price source, and evaluated snapshot.^[1]

Calculate the whole request cost

Start with uncached input tokens, cached input, output tokens, reasoning tokens where billed, and any per-image, audio, video, search, or tool charges. Add retries, rejected outputs, fallback calls, storage, and platform markup. Long prompts can also cross a provider’s context-based rate tier, changing the price of the whole request rather than only the tokens above a threshold.

Then divide total spend by accepted tasks. A model costing half as much per token but requiring twice as many retries has not saved money. An expensive model can be economical when it shortens outputs, avoids escalation, or completes a high-value task correctly on the first attempt.

Define quality in the language of the workflow

Generic intelligence scores can help build a shortlist. Production quality needs a rubric: extracted fields correct, citations support claims, code passes review, format is valid, policy constraints are followed, or a customer response is complete. Weight errors by consequence; a wrong date and a leaked credential should not contribute equally to one average.

Use blinded human comparison for subjective outputs and deterministic graders for facts, schemas, or tests. Calibrate model judges against people before scaling them. Keep segment results because the same model can be excellent on English summaries and weak on a specialized language or document format. Public methodologies such as Artificial Analysis are useful only to the extent that their model versions, prompts, repetitions, and scoring remain inspectable.^[4]

Split latency into the parts users feel

Time to first token determines how responsive streaming feels. Total generation time affects completion. Queueing, retrieval, tool calls, safety checks, and retries add their own delays. Report p50 and p95 because an acceptable median can coexist with a painful tail.

Match the budget to the interaction. A person waiting in chat may need a quick first token, while an overnight research job can trade time for depth. A voice interface is more sensitive to turn-taking delay than a report generator. Do not let one company-wide latency target flatten these differences.

Plot the efficient frontier

Run every candidate on the same representative suite. Plot accepted-answer quality against cost, then encode latency as color or point size. Remove dominated models—those for which another candidate is cheaper, faster, and at least as good. The remaining frontier shows real choices instead of a crowded table.

Choose a point based on the workflow’s error cost and service objective. Repeat with confidence intervals or multiple trials so close points are not overinterpreted. A 0.3-point quality difference may be noise; a large p95 latency difference may be operationally decisive.

Use routing when task difficulty varies

If one model handles most routine traffic and another materially improves hard cases, a router can select between them. RouteLLM reported substantial benchmark-specific savings by learning from preference data. FrugalGPT studied cascades that call additional models only when needed. These studies demonstrate possibility, not a guaranteed percentage for a new workload.^[2][3]

Evaluate the router end to end. Include the classification overhead, wrong-route cost, fallback behavior, and quality of the final response. Preserve a receipt showing which public alias and policy version served each request so operations can explain changes.

Reliability belongs on the same chart

A model that times out, returns invalid tool arguments, or hits rate limits turns theoretical efficiency into user-visible failure. Track request success, schema validity, tool completion, retry rate, and provider health. Calculate cost and latency including failure paths.

Set a fallback policy before incidents. The alternative must satisfy capability and data requirements, not merely be available. Test provider failure and confirm that the receipt and billing describe the model that actually completed the task.

Refresh the decision when the inputs change

Pin model versions where possible and record pricing revisions. Rerun the suite when a provider updates an alias, launches a new model, changes a context tier, or alters rate limits. Use shadow traffic to estimate the impact before changing the default.

The winning architecture may be a portfolio: local for sensitive routine work, a fast hosted default, a reasoning escalation, and a modality specialist. The decision remains coherent if every path is measured against the same accepted-task definition.

Sources and evidence

Primary and authoritative sources used for factual claims. Company research and executive forecasts are labeled as such in the article.

1. 1
   Artificial Intelligence Index Report 2025Stanford HAI · 2025
2. 2
   RouteLLMUC Berkeley, Anyscale, LMSYS · 2024-06-26
3. 3
   FrugalGPTStanford University
4. 4
   Intelligence benchmarking methodologyArtificial Analysis