Ensembling several large language models usually means running all of them at every step and averaging their predictions. That multiplies compute and latency by the number of models. A recent observation makes most of this cost avoidable. Averaging the models' next-token distributions with fixed weights and then sampling gives exactly the same result as first picking one model at random according to those weights and then sampling only from it, so only one model needs to run per token. The weights in that method are fixed for the whole run. The literature on "drowning", where a weak or mismatched model flattens a strong model's confident prediction, shows that fixed weights are the wrong choice. A model that is good on average can still emit noise on particular prompts or particular reasoning steps.
We propose to make the mixture weights depend on the prompt and the text generated so far, while still running exactly one model per token. A small probe attached to whichever model just ran reads that model's hidden state and predicts which model in the pool is most likely to produce the next token well. The next token is then sampled from the model chosen by these predicted weights. The probe is trained to imitate a quantity that the mixture view already defines: each model's posterior "responsibility" for the tokens of correct solutions. Models with near-zero predicted responsibility are dropped for that step. Performance-based model selection then happens per token instead of once per deployment, at close to single-model cost.
Core Claim
The mixture-model ensemble of Rethinking LLM Ensembling from the Perspective of Mixture Models (Fu et al.) samples a token from
$$P(y_t \mid y_{<t}, x) = \sum_{m=1}^{M} \pi_m \, P_m(y_t \mid y_{<t}, x).$$
Here $x$ is the prompt, $y_{<t}$ is the text generated so far, $P_m$ is model $m$'s next-token distribution, and $\pi_m$ are fixed weights. Their key step is to sample a model index from $\pi$ and then a token from that single model. Combined with "lazy" key-value cache synchronization (each model catches up on tokens it skipped only when it is next selected), this runs about 1.78x to 2.68x faster than conventional ensembling.
That equivalence still holds if $\pi_m$ is replaced by $\pi_m(x, y_{<t})$, any function of the prompt and prefix that does not look at the token being sampled. This gives us room to choose the weights per step. We set them as
$$\pi_m(x, y_{<t}) \propto \mathbb{1}\left[g_{m}(h_{t}) \ge \tau\right] g_{m}(h_{t}).$$
In this expression, $h_t$ is the hidden state of the model that produced token $t-1$, $g_m$ is a small learned probe that outputs model $m$'s predicted competence for the next token, and $\tau$ is a pruning threshold.
We claim that this gating keeps per-token cost within a small margin of a single model. We also claim it beats both the static mixture and the best single model, and that the advantage is largest in pools where static ensembling is known to fail.
The second claim matters because Determine-Then-Ensemble (Yao et al.) reports that probability-level ensembling degrades the stronger model when the performance gap exceeds about 10% or when reasoning styles differ. If gating can make such "incompatible" pools useful, it turns a precondition on the whole pool into a decision made per token.
Training the Gate
The training target comes from the mixture view itself. On a training set of correct solutions, we teacher-force each model on the reference trajectory and record $P_m(y_t \mid y_{<t}, x)$ for every model at every position. Each model's responsibility for token $t$ is $r_{m,t} \propto P_m(y_t \mid y_{<t}, x)$, which is the expectation-step quantity of a mixture model with a uniform prior. We train one linear or two-layer probe per (source model, target model) pair to predict $r_{m,t+1}$ from $h_t$. The probe must predict one step ahead because the choice of model for token $t+1$ has to be made before that model runs.
This offline pass costs one teacher-forced pass per model over the training data. Inference never requires running every model.
Two design questions get explicit study. The first is the cost of switching models. Every switch forces the newly chosen model to process the tokens it skipped, so we add a switching penalty to the gate and measure the trade-off between accuracy and switch rate. The second is overfitting to the selection data. Can Bayesian Optimization Efficiently Find a Strong Single Expert in Neural Thickets? shows that selecting on a 200-item validation split of MATH500 picked models that performed worse than the base model on test. We therefore train the gate and tune $\tau$ on disjoint splits, and report test-minus-validation gaps throughout.
Relation to Prior Work
Prior work supplies the method and the motivation but leaves the specific gap open.
Fu et al. supply the single-pass sampling mechanism. They use only static, globally tuned weights, and the method requires stochastic sampling rather than greedy decoding. We inherit the sampling requirement and do not claim to remove it.
Yao et al. supply the motivation. Their UNITE method keeps only the top-$k$ tokens from each model, which cuts cross-model overhead to about 10 ms/token. However, it still runs every model at every step, and it imposes the ≤10% gap rule as a one-time filter on the whole pool.
Three routing systems constrain the design. RACER uses conformal risk control to pick an instance-dependent subset of models before generation. It cuts calls by up to 58.6%, but it runs more than one model whenever the subset is larger than one, and its decision is made once per query. JiSi routes using candidate responses, so the generation cost is paid before the gating decision. LLM-Blender runs all 11 candidates before ranking them. Our gate decides per token, from the currently running model's own state, before the next model runs.
Weight merging is the other route to single-model cost: Model soups and Twin-Merging, which adds a learned router that injects compressed expert weights into a shared backbone. Both require the candidates to share a common starting set of weights. Our method requires only a shared vocabulary.
Our approach has one real cost compared with merging. All $M$ models must stay resident in memory. We reduce compute and latency, not memory.
Research Program
We first run a decisive oracle study, and only then build the learned system.
In the oracle study, an "oracle token gate" picks, at each step, the model with the highest teacher-forced responsibility. We compare it against an "oracle query router", which picks the single best model per prompt in hindsight. If oracle token-level gating does not clearly beat oracle query-level routing on correct-solution likelihood and on downstream accuracy, the token-level idea has no headroom, and the project should be redirected toward query-level routing.
If the oracle study shows headroom, we train the probes and evaluate in three pool regimes. All pools share a tokenizer, so we avoid the vocabulary projection that UNITE needs; cross-tokenizer pools are left as an extension. The first regime is a compatible pool within the 10% gap, such as general instruction-tuned models from the same family. The second is a mismatched pool with a gap above 10%. The third is a pool of complementary specialists, such as a math-tuned and a general instruction-tuned variant that share a vocabulary.
We report accuracy alongside milliseconds per token, forward-pass FLOPs including catch-up prefill after switches, switch rate, and peak memory, all on the same hardware for every method.
Baselines
The naive reference point is the single best model chosen on validation, decoded at the same temperature as the ensemble methods and also greedily. It defines 1x cost. Any method that does not beat it at a cost below about 1.2x has failed. This model is also the reference against which RACER reports +5.0% average accuracy on MMLU, which gives a scale for what routing can achieve.
The most direct competitor is static mixture sampling, the method of Fu et al. with its lazy key-value synchronization. Its weights are grid-searched on the same validation split we use for $\tau$. The file reports that this method is comparable to conventional ensembling and 1.78x–2.68x faster. However, no reported absolute GSM8K or MMLU accuracy for this method was found on Lacuna, so we reproduce it. Beating it means higher accuracy at matched cost (within 10% ms/token) in all three regimes, and a clear margin in the mismatched pool.
The strongest accuracy reference is UNITE, which runs all models in parallel. For LLaMA3 and Qwen2 it reports +3.39% on GSM8K and +4.82% on MMLU over the base models (Determine-Then-Ensemble), at 87.78 versus about 75 ms/token. We do not need to beat UNITE's accuracy. Success means matching most of its gain while running one model per step, and beating it in mismatched pools, where its own analysis predicts degradation.
The query-level router is a probe on the prompt-only hidden state that picks one model for the whole response. It uses the same probe architecture, training data, and tuning budget as our gate, so it is a strict ablation of per-token decision-making.
Two further ablations isolate the remaining design choices. The first replaces the responsibility target with a binary "this model's top token matches the reference" target. The second removes pruning by setting $\tau = 0$, which tests whether hard exclusion, and not just reweighting, is what counters drowning.
Why not just…?
Why not just deploy the single best model? It costs 1x and has no moving parts. Published routing and aggregation results show it leaves accuracy on the table. On MixInstruct, the best single model, OpenAssistant, had an average rank of 3.90 versus 3.01 for LLM-Blender. RACER reports +5.0% on MMLU over the best single model. Those gains, however, come at more than 1x cost. Our single-best baseline at matched temperature measures directly whether a 1x-cost ensemble recovers any of that margin.
Why not just filter the pool with the 10% rule and run a static mixture? This is a real threat for compatible pools. In that regime, static weights may already capture most of the gain, and our gate may add little. We expect our advantage to appear in mismatched and specialist pools, where the filter would discard a model that is weak overall but strong on particular steps. The three-regime design tests exactly this. If the gate only ties the static mixture everywhere, the claim fails.
Why not just route once per query? Query-level routing also runs at 1x cost and is far simpler. It is the most serious alternative. The oracle study is designed to settle it before any system is built, and the matched query-router baseline settles it again with learned components. If per-token switching does not beat per-query routing after accounting for switch costs, we will report that and recommend query routing.
Proposed Experiments
We evaluate on GSM8K, MATH500, BBH, ARC-Challenge, and MMLU, the benchmarks used by Fu et al., Yao et al., and RACER. Reasoning tasks are where switching models mid-solution plausibly helps; multiple-choice knowledge tasks are a control where per-token gating should matter little. Each setting is run with three seeds, and confidence intervals are bootstrapped over items.
The decisive checks are as follows. The oracle token gate must beat the oracle query router. The learned gate must beat the tuned static mixture on GSM8K and MATH500 in the mismatched pool, at no more than 1.2x single-model ms/token. The validation-to-test gap of the gate must stay small on MATH500, the benchmark where the Bayesian-optimization search was shown to overfit.
We also analyze when switches occur, for example at arithmetic steps or at format tokens. This tests whether the gate learns the style-incompatibility effect that Yao et al. identified at the level of whole models.
References
- Rethinking LLM Ensembling from the Perspective of Mixture Models
- Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling
- Can Bayesian Optimization Efficiently Find a Strong Single Expert in Neural Thickets?
- RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models
- Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scale
- LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging