When to Vote, When to Rewrite: Disagreement-Guided Strategy Routing for Test-Time Scaling

Zhimin Lin, Yixin Ji, Jinpeng Li, Yu Luo, Dong Li, Junhua Fang, Juntao Li, Min ZhangarXiv 2026 · arXiv

TL;DR

Instead of uniformly throwing massive computational resources at every problem during inference, this paper introduces a training-free framework that dynamically routes instances to different test-time scaling strategies based on model disagreement. By using lightweight processing for consistent answers, majority voting for moderate variance, and prompt rewriting for severe disagreements, it boosts accuracy by 3%–7% while significantly cutting down on computational overhead.

What Problem Does This Paper Address?

Large Reasoning Models (LRMs) excel at complex tasks but remain unreliable on highly challenging edge cases. To improve output quality, researchers often turn to test-time scaling—allocating extra compute at inference time through methods like repeated sampling, self-correction, or tree search.

However, existing approaches typically apply a single scaling strategy uniformly across all problems. This leads to two major inefficiencies: wasting compute on easy instances that don't need it, and hitting diminishing returns on hard problems. Crucially, the authors discover a performance trade-off between common strategies: prompt rewriting helps models escape incorrect reasoning paths on hard problems, but it actually degrades performance on easy problems by introducing unnecessary errors or altering semantics.

Figure 1: The effect of rewriting and majority voting (6 samplings) for Qwen3-8B across instances of different difficulty levels. Easy instances are defined as those correctly answered in a single sampling, while hard instances are those incorrectly answered in a single sampling.

The challenge is to figure out not just how much compute to allocate, but which inference strategy fits a given problem's difficulty level.

Methods

The authors propose a training-free, multi-stage routing framework guided by a Minimal Disagreement Detector (MDD). The core idea is to use the variance of early model outputs as a proxy for uncertainty and difficulty, routing instances accordingly through three distinct stages:

Figure 2: This figure illustrates the overall workflow of our method, where MDD, NDS, MDS, and SDS stand for Minimal Disagreement Detector, No Disagreement Samples, Minor Disagreement Samples, and Severe Disagreement Samples, respectively. The four model blocks in the figure all refer to the same model.
  1. Disagreement Filter (Stage 1): The model samples the problem twice. If the answers match, it is classified as a No Disagreement Sample (NDS) and the answer is returned immediately, preventing redundant computation.
  2. Vote Resolve (Stage 2): If the first two samples disagree, the model generates two more samples. If these subsequent samples match, it indicates moderate uncertainty—a Minor Disagreement Sample (MDS)—which is resolved using standard majority voting across all four available samples.
  3. Rewrite & Rethink (Stage 3): If the outputs remain highly inconsistent after multiple checks, it is flagged as a Severe Disagreement Sample (SDS). The framework prompts the model to rewrite and reformulate the surface expression of the question (clearing up potential structural or hidden ambiguities) before sampling a final answer.

Datasets

The framework was rigorously evaluated across two domains:

  • Mathematical Reasoning: Seven widely recognized benchmarks including GSM8K, Math500, Gaokao2023en, Olympiadbench, AMC23, AIME24, and AIME25.
  • Code Generation (for generalization testing): HumanEval, HumanEval+, MBPP, and MBPP+. For coding tasks, disagreement was assessed via functional execution-based equivalence (running test cases) rather than exact string matching.

Key Insights and Ideas

  • Disagreement as a Free Signal: Output variance under stochastic sampling serves as an excellent, training-free indicator of problem difficulty and model uncertainty. High consensus strongly correlates with correctness, while a flat answer distribution signals a high likelihood of failure.
  • The Rewriting Paradox: Prompt rewriting acts as a double-edged sword. It simplifies complex problems by making implicit constraints explicit, which helps models break out of repetitive, flawed reasoning loops. However, on simple questions, rewriting often over-compresses text and strips away vital information, causing the model to fail on tasks it originally answered correctly.
  • Compute-Accuracy Efficiency: By filtering out easy problems in the very first stage, the framework frees up the compute budget, allowing resources to be concentrated precisely where the model is struggling.

Key Results

  • Consistent Accuracy Gains: Across three base models (Qwen3-4B, Qwen3-8B, and DeepSeek-R1-Distill-Llama-8B), the routing method consistently outperformed baselines (like standard majority voting and SCoP), yielding a 3% to 7% absolute accuracy improvement on average.
  • Substantial Compute Savings: Despite achieving higher accuracy, the proposed method used significantly fewer total samplings. On datasets like Math500 and Olympiad, it consumed only 40% to 60% of the maximum sampling budget compared to fixed-allocation strategies.
The number of samplings used by different methods across benchmarks.
  • Cross-Domain Generalization: The method successfully translated to code generation tasks. For instance, it boosted the average performance of Qwen2.5-Coder-7B-Instruct from 80.4% to 83.5% without any task-specific tuning.

Limitations and Future Work

  • The "False Agreement" Blindspot: The framework relies entirely on output disagreement to detect difficulty. If a model confidently and consistently arrives at the wrong conclusion (a false agreement), the router will mistakenly classify it as an easy problem and exit early.
  • Brittle Rewriting: While routing minimizes harmful rewrites, it cannot completely eliminate cases where prompt reformulation accidentally introduces incorrect assumptions or drops critical constraints.
  • Limited Strategy Pool: The current implementation only routes between voting and rewriting. The authors note that expanding the framework to route toward more sophisticated test-time scaling strategies, such as tree-search or verifier-based methods, remains an open and promising direction.

Summary updated

Abstract

Large Reasoning Models (LRMs) achieve strong performance on mathematical reasoning tasks but remain unreliable on challenging instances. Existing test-time scaling methods, such as repeated sampling, self-correction, and tree search, improve performance at the cost of increased computation, yet often exhibit diminishing returns on hard problems. We observe that output disagreement is strongly correlated with instance difficulty and prediction correctness, providing a useful signal for guiding instance-level strategy selection at test time. Based on this insight, we propose a training-free framework that formulates test-time scaling as an instance-level routing problem, rather than allocating more computation within a single strategy, dynamically selecting among different scaling strategies based on output disagreement. The framework applies lightweight resolution for consistent cases, majority voting for moderate disagreement, and rewriting-based reformulation for highly ambiguous instances. Experiments on seven mathematical benchmarks and three models show that our method improves accuracy by 3% - 7% while reducing sampling cost compared to existing approaches.
BibTeX
@misc{lin2026when,
  author = {Zhimin Lin and Yixin Ji and Jinpeng Li and Yu Luo and Dong Li and Junhua Fang and Juntao Li and Min Zhang},
  title = {{When to Vote, When to Rewrite: Disagreement-Guided Strategy Routing for Test-Time Scaling}},
  year = {2026},
  eprint = {2604.26644},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.26644},
}
Figure