Ensemble Only Where the Choice Changes the Answer

Combining several language models at generation time, averaging their next-word predictions, makes answers more accurate. It also multiplies the serving cost by the number of models. Recent methods cut that cost by combining models only at the words where they disagree. Disagreement, though, is a weak proxy for what matters. Two models can disagree on a comma or a synonym, and paying for a multi-model step there buys nothing. They can also mildly agree on a wrong number that decides the whole solution, and the ensemble is never consulted. We propose to decide when to ensemble by predicting the consequence of the choice at each position: how likely it is that swapping the lead model's word for the ensemble's word would change the final answer.

The prediction comes from counterfactual labels computed offline. We force the alternative word at a position, let generation continue, and record whether the outcome flips. A small predictor is then trained on these labels and reads only the lead model's internal states, so it can decide at inference time whether to wake the other models at all. This borrows the core move of token-importance research, measuring a token's importance by what happens when you remove or replace it. It applies that move to generated decisions rather than input words, and uses it to budget ensemble compute.

Core Claim

The closest existing method is SAFE (When to Ensemble). SAFE lets one model draft and has the others verify. It ensembles only at positions where the models' vocabularies line up and the models lack consensus. On math, models agree on over 95% of tokens, and SAFE ensembles fewer than 1% of them. On general reasoning such as BBH (Big-Bench Hard), it still ensembles 10–15%.

SAFE's gate asks "do the models differ here?" Ours asks "would the difference propagate to the answer?" We claim that a consequence-gated ensemble traces a strictly better curve of accuracy against ensemble cost than consensus gating, for two reasons. On reasoning tasks it skips disagreements that do not matter. On math it catches low-disagreement pivot points that consensus gating passes over. Because the gate runs on the drafter alone, it also cuts how often the verifier models must be invoked, not merely how often their distributions are averaged.

The gate works under greedy decoding. That matters because the mixture-model shortcut of Rethinking LLM Ensembling from the Perspective of Mixture Models runs one model per token by sampling a model at random with fixed weights, and so cannot handle greedy decoding. Its authors name input-dependent token-level routing as an open problem, and this proposal addresses it.

The target quantity is defined as follows. Let $y_t$ be the drafter's token at step $t$, let $\hat y_t$ be the token the full ensemble would choose, and let $A(\cdot)$ be the final answer reached by greedy continuation from a prefix. The consequence of position $t$ is

$$c_t = \Pr\big[A(x, y_{<t}, \hat y_t) \neq A(x, y_{<t}, y_t)\big],$$

where $x$ is the prompt. The probability is taken over sampled continuations when decoding is stochastic, and $c_t$ is a 0/1 indicator under greedy decoding. We ensemble at $t$ when the predicted $\tilde c_t$ exceeds a threshold $\tau$. Sweeping $\tau$ traces the cost–accuracy curve.

Research Program

Labeling. We use the three heterogeneous 7–8B models from SAFE: InternLM3-8B, Qwen2.5-7B and EXAONE-3.5-7.8B. We run the full per-token ensemble alongside the drafter on training splits of GSM8K and MATH, and on held-out BBH tasks that are disjoint from our evaluation tasks.

At every aligned position where $\hat y_t \neq y_t$, and at a random sample of agreement positions, we roll out both branches and record whether the answer changes. For agreement positions, the alternative branch substitutes the ensemble's second choice, so that "silent pivots" can be labeled. This is the leave-one-out substitution of Is BERT Really Robust? and On the Word Alignment from Neural Machine Translation. Those papers measure influence by the output drop after deleting or masking an input token. We move it from input tokens to the outcome of a whole generated trajectory. The labeling is expensive, roughly two rollouts per labeled position, but it is paid once and offline.

Predictor. We distill the labels into a lightweight head on the drafter's hidden states. The recipe follows Attribution-Driven Adaptive Token Pruning for Transformers, which distills expensive Integrated Gradients scores into a cheap token-saliency predictor. The architecture follows TokenButler, which shows that a head of about 1% of the model's parameters can predict token importance from early hidden states. The head's inputs deliberately include the drafter's entropy and top-two margin. The question is then whether the counterfactual labels teach it anything beyond those cheap signals.

Serving. The drafter decodes alone while $\tilde c_t < \tau$. The verifiers' key-value caches, their stored per-token context, fall behind during this stretch. When the gate fires, the verifiers catch up in a single batched pass over the skipped span, similar to how speculative decoding verifies a draft block. They then produce the ensemble distribution at $t$, sharpened as in SAFE.

Total verifier FLOPs are unchanged by this design, because every token is eventually processed. What falls is the number of sequential, memory-bound verifier invocations and the number of averaging and replacement events. We therefore report wall-clock throughput alongside invocation counts. We do not claim FLOP savings.

Baselines

The naive reference points are the single best model and the full per-token ensemble (UniTE, the averaging method SAFE builds on), run with identical prompts and greedy decoding. They bound cost from below and above.

The strongest competitor is UniTE+SAFE, which reports 77.6% accuracy on MATH500 against 59.6% for UniTE alone (When to Ensemble). We rerun it on the same three models and hardware, with the same probability sharpening and cache handling. We also sweep its consensus threshold, so it traces a full cost–accuracy curve rather than a single point. Beating it means one of two things. Either we match 77.6% on MATH500 with fewer verifier invocations, or we exceed it at equal invocations. On BBH we want higher accuracy at an ensembling rate well below its 10–15%. We found no reported BBH accuracy for SAFE on Lacuna, so both methods are measured there under our own runs.

For throughput we also compare against Speculative Ensemble, which verifies drafted tokens against the joint ensemble distribution and reports 1.34x–1.85x speedups over standard weighted ensembling. We use the same models and draft length.

The key ablations isolate the counterfactual label. The first trains the same head with identical inputs and capacity, but on disagreement-only labels. The second is an entropy/margin gate with no learned head. The third is an attention-based importance gate in the style of STEP (Less Token, More Signal). All gates get the same threshold-tuning budget on a shared validation split.

Why not just…?

Why not just gate on the drafter's entropy or top-two margin? This is the real threat. Uncertain positions may simply be the consequential ones. Agreement-Based Cascading reports that single-model confidence is poorly calibrated as a deferral signal. Entropy is also local by construction and cannot see whether a choice propagates. Still, we will not argue this away. The learned head receives entropy and margin as inputs, and the entropy-only gate is a matched baseline. If the two curves coincide within noise on BBH and MATH500, the central claim is false. In that case the contribution shrinks to a measurement: how often consequential positions are low-entropy.

Why not just tune SAFE's consensus threshold? A stricter threshold lowers cost, but it can only reorder positions by how much the models disagree. It cannot trigger at the silent pivots where models agree yet the second choice would change the answer. We measure directly what fraction of answer-flipping positions have consensus. If that fraction is near zero and threshold-swept SAFE matches our curve, consequence gating adds nothing over consensus.

Why not just use attention mass as the importance signal? Attention is free to compute, and STEP shows that about 20% of tokens carry about 80% of attention weight. But attention measures how much tokens attend to one another, not their marginal effect on the output. Attribution-Driven Adaptive Token Pruning found that attention-based pruning loses to attribution-distilled pruning on GLUE. Attention also scores a token's influence as context, not whether choosing it differently would matter. The attention-gate ablation tests this directly.

Why not route whole queries to one model instead? Query-level routing pays for an entire sequence at the capacity of the chosen model. Both LLM-Blender and JiSi find that aggregation beats the single best model. That is the gain token-level gating tries to keep at lower cost.

Proposed Experiments

We evaluate on GSM8K, MATH500, BBH (held-out tasks), ARC-Challenge, MMLU and HumanEval. We report accuracy, or pass@1 for HumanEval, against four cost measures: verifier invocations per 100 tokens, fraction of ensembled positions, tokens per second, and peak GPU memory. All numbers are averaged over three seeds of the labeling split.

First decisive check. Before any serving work, we test ranking quality on held-out labeled positions. We compute precision–recall for predicting answer flips, comparing the learned head, entropy and disagreement. If the head does not clearly beat entropy here, we stop.

Second decisive check. We compare full cost–accuracy curves against threshold-swept SAFE. The prediction is a gain concentrated on BBH-style tasks, where SAFE's ensembling rate is high. On math the prediction is mainly fewer verifier invocations rather than higher accuracy, because SAFE already ensembles under 1% of math tokens.

Third check: transfer. A head trained on math labels is evaluated on BBH and HumanEval. This shows whether the head learns general signs that a choice matters, or task-specific surface cues. Poor transfer would limit the method to settings where offline labeling is affordable.