When Strong Reasoners Join the Pool: Small Union Premiums and Costly Runaway Generations in LLM Routing

Teng-Ruei ChenarXiv 2026 · arXiv

In the quest to optimize LLM deployments, "routing"—the practice of choosing the best model for a specific query—is often hailed as a silver bullet for balancing cost and quality. However, a new technical report by Teng-Ruei Chen suggests that as our "reasoning" models get stronger, the traditional metrics we use to judge routers might be giving us a misleading picture of the available gains.

TL;DR

As LLMs become "strong reasoners," the extra accuracy gained by combining multiple models (the "union premium") shrinks significantly, dropping from ~3.5% to less than 0.6% on the MATH-500 benchmark. Furthermore, the study reveals that "runaway generations"—rare instances where models get stuck in infinite loops—can consume up to 60% of a model's computational budget, making robust decoding caps more critical than complex routing logic.

What Problem Does This Paper Address?

Most routing benchmarks compare a router against a "hindsight oracle." This oracle is credited with a win if any model in the pool gets the answer right. But there’s a catch: under stochastic decoding (where models can give different answers to the same prompt), this oracle conflates two very different things:

  1. Selection Headroom: The gain from picking the single best model for a specific query.
  2. Union Premium: The gain from calling multiple models and keeping any correct answer.

The author argues that the "Union Premium" is not a "routing gain" because it requires multiple expensive calls. If we don't separate these two, we overstate how much a real-world router (which usually commits to one model) can actually improve performance. This paper investigates how these dynamics change when high-performing reasoning models like Llama-3.3-70B and NVIDIA’s Nemotron series enter the pool.

Methods

The study employed a rigorous, "fixed-protocol" approach to ensure reproducibility:

  • Hardware: A single NVIDIA B300 (Blackwell) GPU.
  • Protocol: 30 seeded samples were generated for every problem in the dataset across three models.
  • Models: Llama-3.3-70B-Instruct, NVIDIA Nemotron Nano 9B v2, and Nemotron Super 49B v1.5.
  • Verification: All 45,000 responses were scored using a pinned version of the Math-Verify tool.
  • Audit Trail: The author used an append-only, hash-chained ledger to record every generation attempt, preventing "cherry-picking" of results.

Datasets

The evaluation focused exclusively on MATH-500, a subset of the challenging MATH dataset. This choice allows for direct comparison with a companion study that used 11 smaller, non-reasoning models on the same problems.

Key Insights and Ideas

  • The Shrinking Premium: In previous studies with weaker models, the "union premium" was a healthy 3.54 percentage points (pp). With strong reasoners, it collapses to just 0.22–0.59 pp. Because these models are already so accurate (96%+), there are very few "disagreements" left for a union to resolve.
  • Resampling is the New Routing: One of the most striking findings is that simply calling the smallest strong model (Nemotron Nano 9B) three times yields the same accuracy as calling three different models once. This suggests that "test-time compute" (more samples) is often more effective than "model diversity."
  • The "Costly Tail" of Runaway Generations: While only 31 out of 45,000 responses hit the 65,536-token cap, those few responses accounted for a staggering 60.2% of the 70B model's computational proxy (output-length-squared). Most of these were "periodic tails"—the model getting stuck in a repetitive loop.
Overview of the study's findings, highlighting the small union premium and the high cost of capped responses.

Key Results

  • Accuracy: Both reasoning models (Nano 9B and Super 49B) reached 96.1% accuracy, significantly outperforming the 76.2% baseline of the standard 70B model.
  • Oracle Decomposition: For the three-model pool, the total "oracle gap" over the best single model was 1.86 pp. However, 1.27 pp of that was "selection headroom" (potentially capturable by a router), while only 0.59 pp was the "union premium" (requiring multiple calls).
  • Efficiency: Nemotron Nano 9B achieved the same accuracy as the Super 49B model but required 4.7x less GPU time per 1,000 correct answers.
Fig. 3. Ceilings and premiums per pool (Q1 = 70B + Super, Q2 = Nano + Super, Q3 = 70B + Nano, Q4 = all three). (a) Single-commit ceiling ¯ˆR (one call, clairvoyant), product union ¯ˆU⊥(all models, gold-based selection) and the premium between them; the grey band is the range, over the 30 sample indices, of an oracle computed from one recorded sample per model. (b) Gain over the best single model, split into empirical plug-in selection headroom ( ¯ˆR minus the best model’s accuracy) and union premium ( ¯ˆU⊥−¯ˆR), in percentage points.

Limitations and Future Work

  • Ceiling Effects: MATH-500 is becoming "too easy" for these models, leaving little room to measure further gains. The author suggests future work should target harder benchmarks like GPQA-Diamond.
  • Gold-Based Selection: The study uses "gold-based selection" (knowing the right answer) to calculate the union premium. In production, you'd need a "verifier" or "reward model," which adds its own cost and error.
  • Single Benchmark: The results are specific to mathematical reasoning; routing dynamics might look very different for creative writing or coding.
  • Llama-3.1-8B Failure: The planned inclusion of Llama-3.1-8B was aborted because it hit the token cap so frequently that it became too expensive to finish the run, highlighting that even "small" models can be compute-hogs if they lack robust stopping criteria.

Summary updated

Abstract

Routing benchmarks often judge a router against a hindsight oracle that counts a query as solved whenever any model's recorded answer is correct, but under stochastic decoding that oracle mixes two different objects: the clairvoyant ceiling of committing to one model per query and the premium of calling several models and keeping any correct answer. A companion study measured this premium at 3.5 percentage points (pp) for 11 small open models on MATH-500; how large it is once current reasoning models enter the pool was unknown. We measure it with a fixed-protocol, pre-committed generation on a single NVIDIA B300: all 500 MATH-500 problems, 30 seeded samples per problem, Llama-3.3-70B-Instruct and NVIDIA's Nemotron Nano 9B v2 and Nemotron Super 49B v1.5 (a NAS derivative of the same 70B) under vendor sampling, a 65,536-token cap, a pinned Math-Verify scorer, and an append-only attempt ledger, for 45,000 scored responses. The two reasoning models both reach 96.1% accuracy against 76.2% for the 70B baseline. The empirical product-coupling premium is 0.215-0.588 pp across all two- and three-model pools; the empirical plug-in selection headroom over always calling Nano is 1.27 pp; and resampling Nano three times gives practically the same empirical any-correct rate as calling all three models once. Only 31 responses reached the cap, 25 of them with exactly periodic tails, yet the capped responses account for up to 60.2% of a model's output-length-squared proxy, and long generations made a fourth planned model, Llama-3.1-8B, too slow to finish. The premium is far below the companion's 3.5 pp, but models, pool size and protocol changed together; an oracle gap should therefore be reported with the pool, the protocol, the samples per cell and the decoding budget that produced it.
BibTeX
@misc{chen2026when,
  author = {Teng-Ruei Chen},
  title = {{When Strong Reasoners Join the Pool: Small Union Premiums and Costly Runaway Generations in LLM Routing}},
  year = {2026},
  eprint = {2611.04098},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04098},
}
Figure