Predictable performance is suboptimal, and what it means for model routing

Guanhua Zhang, Moritz HardtarXiv 2026 · arXiv

TL;DR

If you can predict a model's errors better than the model itself can, that model is likely suboptimal. This paper proves that the advantage of an external performance predictor is theoretically bounded by the model's "room for improvement" (suboptimality) and the predictor's own size, showing that as models are fine-tuned and improved, the value of complex routing systems over simple self-confidence diminishes.

What Problem Does This Paper Address?

In the world of Large Language Models (LLMs), "routing" is a popular strategy to balance cost and quality. The idea is simple: use a cheap, "weak" model for easy queries and escalate to an expensive, "strong" model only when the weak one is likely to fail.

To do this well, we need to predict performance: will Model A get this right? Most researchers train separate, external "assessors" or "routers" to make this call. However, the authors of this paper ask a fundamental question: Where does this predictive power come from? If an external 4B parameter model can tell that a 7B model is going to fail, why didn't the 7B model "know" that itself? The authors investigate whether the success of these routers is simply a symptom of the underlying models being poorly calibrated or suboptimal for the task at hand.

Methods

The authors establish a rigorous theoretical framework to compare external predictors against a baseline of self-confidence (the probability a model assigns to its own chosen answer).

  1. Recalibration & Exact Transfer: They show that any gain a predictor has in "correctness log loss" (predicting if an answer is right) can be translated exactly into a reduction in the model's "answer log loss" (the standard training objective) through a process called recalibration.
  2. The Suboptimality Bound: Using the concept of a "scaling floor" (the best possible loss for a given parameter budget), they derive a bound: a predictor's advantage over self-confidence is at most the model's suboptimality plus the "capacity gain" provided by the predictor's extra parameters.
  3. Cascading Analysis: They extend this logic to two-model cascades, where a router decides whether to escalate a query. They prove that a router's advantage over confidence-based routing is similarly limited by the weak model's suboptimality.
  4. Experimental Validation: They tested 16 LLMs (7B to 12B range) across four benchmarks. They trained separate 4B-parameter predictors (based on Qwen3-Embedding) for every model-task pair and compared their performance before and after task-specific fine-tuning.

Datasets

The study utilized four diverse benchmarks to evaluate model performance and predictability:

  • WinoGrande: A benchmark for commonsense reasoning (binary choice).
  • CommonsenseQA: Multiple-choice questions requiring world knowledge.
  • GSM8K: Grade school math word problems (requires multi-step reasoning).
  • MMLU-Pro: A more robust and challenging version of the massive multi-task language understanding benchmark.

Key Insights and Ideas

  • Predictability is a "Check Engine" Light: If an external predictor significantly outperforms a model's own self-confidence, it serves as a certificate that the model is suboptimal. There is "leftover" information that the model failed to incorporate into its own weights.
  • The Diminishing Returns of Routing: As we fine-tune models to be better at specific tasks, they naturally become more "self-aware" of their limitations. This means the "excess predictability" available for a router to exploit shrinks.
  • Capacity Matters: A predictor's advantage isn't just about better logic; it's also about the extra parameters it brings to the table. A very small predictor (e.g., 0.6B) adds almost nothing beyond simple calibration, while larger ones (4B, 8B) can find more complex patterns in the model's errors.
  • Self-Confidence is a Strong Baseline: For well-optimized models, simply looking at the probability of the top-1 token is a surprisingly hard-to-beat heuristic for routing.

Key Results

The empirical results strongly support the theoretical bounds.

  • Correlation with Suboptimality: Across all four tasks, there was a strong positive correlation between how suboptimal a model was and how much an external predictor could improve upon its self-confidence. As seen in Figure 1, the "Excess Predictability" (E) scales almost linearly with "Empirical Suboptimality."
Figure 1: Predictors have a larger advantage over self-confidence on more suboptimal models. Each point is a released (orange) or finetuned (blue) model. Empirical suboptimality is the gap to the best observed answer log loss on each task. Excess predictability E is the reduction in correctness log loss achieved by a separately trained 4B predictor over self-confidence. Colors identify training stage; shapes identify the release type. Lines are least-squares fits; r is Pearson correlation and p+ its one-sided p-value for positive correlation. See Section 3.4.1 for experimental details.
  • Fine-tuning Erases the Advantage: When models were fine-tuned on the task, their suboptimality dropped, and the advantage of the 4B predictor plummeted. In 55 out of 59 model-task pairs, fine-tuning reduced the "excess predictability."
  • Routing Gains: The "routing advantage" (the accuracy gain from using a learned router vs. confidence-based routing) followed the same trend. More suboptimal models showed larger routing gains, but these gains diminished significantly after fine-tuning.
Figure 4: More suboptimal models have larger routing advantages.

Limitations and Future Work

  • The "Strong Model" Variable: The current theory focuses heavily on the "weak" model's suboptimality. However, in a real-world cascade, the router also needs to know if the strong model will get the answer right. The authors note that future work should characterize the strong model's contribution more deeply.
  • Upper Bounds vs. Realized Gains: The theory provides an upper bound on how much a predictor can help, but it doesn't guarantee that a specific training method will reach that bound.
  • Log Loss vs. Accuracy: While the theory relies on log loss (which is mathematically convenient), practitioners often care more about discrete accuracy. While the trends hold for both, the mapping isn't always 1:1.
  • Beyond Cascades: The study focuses on a specific routing setup (escalation). Exploring how these insights apply to more complex "mixture-of-agents" or multi-step routing remains an open area.

Summary updated

Abstract

Large language models (LLMs) differ in capabilities and costs, motivating routing methods that adaptively choose a model for each query. These methods rely on performance prediction: given a query, estimate whether a model will answer it correctly. Most routing methods train separate predictors of candidate models' performance to guide model selection. Our work shows that performance prediction only beats a simple baseline when the candidate model is suboptimal on the task or the performance predictor adds significant capacity relative to the candidate model. Specifically, we take a model's self-confidence, the probability it assigns to its chosen answer, as a baseline and study how much a performance predictor can improve on it. First, we show that any performance predictor that outperforms a model's self-confidence can be used to improve the model's loss: the predictor's advantage transfers exactly in log loss through recalibration. Second, accounting for the predictor's additional parameters, we bound its advantage by the model's suboptimality plus the gain from those parameters. Third, we extend our theoretical results to two-model cascades, which route queries to a more capable model when the first model is predicted to fail. Experiments with 16 language models on four benchmarks show that predictors gain a larger advantage over self-confidence on more suboptimal models. This advantage shrinks as models improve through finetuning. Cascading exhibits the same two patterns: predictor-based routing offers larger gains over confidence-based routing for more suboptimal models, and these gains diminish after finetuning.
BibTeX
@misc{zhang2026predictable,
  author = {Guanhua Zhang and Moritz Hardt},
  title = {{Predictable performance is suboptimal, and what it means for model routing}},
  year = {2026},
  eprint = {2610.31428},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.31428},
}
Figure