Stopping a Weighted Ensemble Once the Rest Can No Longer Change Its Vote

Strong systems for specialized text, such as clinical-trial inference, are often built as weighted ensembles. Several models, some pre-trained on biomedical literature and some general-purpose, each cast a vote. Each vote is scaled by how reliable that model proved on held-out data. These ensembles win shared tasks, but they run every model to full depth on every input. Their authors name that cost as the main obstacle to deployment. Early exiting is the standard fix for wasted inference compute: a network stops partway through its layers once an intermediate prediction looks confident enough. So far it has been applied inside single models, judged by that model's own confidence. This proposal brings the two together. We would run the members of a domain ensemble cheapest-first. Each member can cast provisional votes from its intermediate layers. The whole ensemble stops as soon as the evidence indicates that the members and layers not yet run cannot overturn the current weighted vote.

The key change is the target the stopping rule serves. Existing early-exit rules ask "is this model confident?" We ask "would finishing the computation change the ensemble's answer?" That question has a safe, exact version: stop when the leading class's margin exceeds all the vote weight still outstanding. It also has a learned, more aggressive version that predicts the chance of a vote flip. We would test both on clinical and biomedical inference benchmarks. The member pool would mix domain-pretrained encoders, a domain-pretrained sequence-to-sequence model, and a 7B generative model. The success criterion is compute: most of the full ensemble's accuracy, faithfulness, and consistency at a fraction of its cost.

Core Claim

A performance-weighted ensemble of heterogeneous domain specialists, extended with intermediate exits, can match its own full-execution predictions on most inputs while running only a few cheap members partway. The stopping signal that achieves this is not any member's softmax confidence. It is an estimate of whether the outstanding vote mass can still flip the weighted decision. We further claim that domain pre-training matters for where in the network the provisional votes become reliable. Domain-pretrained members should produce stable, trustworthy early-layer votes on domain text sooner than general members of the same size. That makes them disproportionately valuable as the cheap front of the queue.

Relation to Prior Work

Three papers motivate the gap by exposing the cost problem. iML at SemEval-2024 Task 2 combines SciFive, a T5 model pre-trained on biomedical text, with a zero-shot 13B generative model. It weights each member's vote by its accuracy on a practice set and reaches 70.27 Macro F1 on the NLI4CT test set. ABCD at SemEval-2025 Task 9 goes further and weights each model's vote per class by its validation F1. Both execute every member fully on every input, and ABCD explicitly names latency as the limitation. We keep their weighting schemes and remove the full-execution assumption.

Three papers supply the method and each stops short of our setting. BEEM treats the exit heads of a single backbone as weighted experts, either FLOP-weighted or accuracy-weighted. It accumulates their score only while adjacent exits agree, reaching 1.9x to 2.1x speedup on GLUE. Its ensemble is confined to one model, on general benchmarks. Early Exit Ensembles for Uncertainty Quantification trains diverse exit heads and averages them uniformly, but never stops early. Compute-Aware Mixture-of-Agents halts multi-agent aggregation when agents agree and recovers from a weak pool member. Its agreement is unweighted, its pool is general-purpose, and its weighted routing step was formulated but not implemented. Our design differs from all three on a specific point: the stopping question is posed against the weighted vote of heterogeneous models, not against one model's internal state or against raw agreement.

Two lines of work constrain the design. Rethinking Calibration for Early-Exit Neural Networks proves that calibrating each exit separately cannot detect futile samples. These are inputs that no layer will get right, so they waste the full network. Our flip criterion partly sidesteps this problem. A sample whose answer will not change gets stopped even if it is wrong, because further compute cannot help it. CeeBERT and DAdEE show that exit thresholds fit on one data distribution break on another. This warns us that vote weights and flip predictors fit on the NLI4CT practice split may not transfer to the test split, and we measure that gap directly.

The domain pre-training evidence frames the application regime. BioBART shows gains from continued pre-training on PubMed, but also a gap when moving to clinical text. Edinburgh Clinical NLP at SemEval-2024 Task 2 shows that an adapted 7B model reaches an NLI4CT average score of 0.7862, against 0.8328 for GPT-4.

Mechanism

Members are ordered by cost. Each member $m$ exposes exits at depths $d$, and each exit has a vote weight $w_{m,d,c}$ for class $c$. We fit these weights on labeled development data in the per-class style of ABCD, extended to depth. A member's later exit replaces its earlier one, so each member holds at most one live vote. After evaluating some set of members and exits, the current weighted score for class $c$ is $V_c$. Let $R$ be the largest total weight that any single class could still gain from all unevaluated members and deeper exits. Then the exact stopping rule is

$$ V_{(1)} - V_{(2)} > R, $$

where $V_{(1)}$ and $V_{(2)}$ are the top two class scores. When this holds, the full ensemble provably returns the current leader. One subtlety must be handled with care. A deeper exit can retract an earlier provisional vote, so $R$ must include the possible withdrawal of each live provisional vote as well as new weight.

The exact rule is conservative. The learned rule replaces it with a small classifier trained on development data. Its inputs are the normalized margin, the remaining mass $R$, cross-member agreement, BEEM-style within-member stability, and member identity. Its target is whether the full ensemble's final answer differs from the current leader. The ensemble stops when the predicted flip probability falls below a threshold $\epsilon$, and sweeping $\epsilon$ traces a compute–fidelity curve.

Research Program

We would build the pool from published components. The cheap front consists of a biomedical encoder or SciFive with attached exit heads trained by self-distillation in the style of Be Your Own Teacher. Next comes BioBART with exits on its decoder classifier. A 7B generative model at the back casts a single final vote, using the Edinburgh adapter recipe. For the domain question we would add size-matched general counterparts: BART for BioBART, and a general T5 for SciFive. These let us measure how early in the network each member's provisional votes become reliable, once with domain pre-training and once without.

The analysis has three parts. First, we would trace the compute–fidelity and compute–accuracy curves for the exact and learned rules. Second, we would measure which members the stopping rule actually skips, and how often the expensive generative member is needed. Third, we would test whether early stopping concentrates errors on NLI4CT's intervention-based faithfulness and consistency subsets. These subsets are where domain models are already weakest, so this check matters most.

Baselines

The naive reference points are two. The first is the single best member run at full depth. The second is the full weighted ensemble run on every input, which serves as both the accuracy ceiling and the fidelity reference. On NLI4CT, the closest published full ensemble is iML's performance-weighted ensemble at 70.27 Macro F1. The strongest published single adapted open model reports a 0.7862 average score, and GPT-4 reports 0.8328 (Edinburgh). On PubMedQA, the published reference is fine-tuned BioBERT at 68.1% accuracy, against 55.2% for majority guess and 78.0% for humans (PubMedQA). We would re-run iML-style global weighting and ABCD-style per-class weighting on our own pool so that all comparisons share members, splits, and tuning data.

The strongest competitor is a plain confidence cascade. It runs the cheapest member and returns its answer if its temperature-calibrated confidence (The Right Tool for the Job) clears a threshold, and otherwise moves to the next member. It is strongest because it captures most of the obvious savings with no ensemble machinery. No reported result was found on Lacuna for such a cascade on NLI4CT or PubMedQA, so we would build it from the same members with the same exits, tuned on the same development split with the same hyperparameter budget. The second-strongest competitor is BEEM applied to the best single domain member. This tests whether multi-model voting adds anything over within-model weighted exiting. BEEM reports 1.9x to 2.1x speedup on GLUE, but no reported result was found on Lacuna for BEEM on biomedical NLI.

Beating these baselines means one of two things on the NLI4CT and PubMedQA test sets. Either we reach higher Macro F1 and consistency than the cascade at equal measured FLOPs and wall-clock time, or we match the full ensemble's Macro F1 within one point at under half its compute. Three ablations isolate the key ingredients. The first removes provisional intermediate votes, so members vote only at full depth. The second replaces the flip target with a correctness target, which recovers a meta-classifier like the one in Rethinking Calibration. The third replaces learned weights with uniform weights, which recovers Compute-Aware MoA-style agreement.

Why not just…?

Why not just use a calibrated confidence cascade? This is the real threat. If the cheap domain encoder is right whenever it is confident, the cascade will match us and the voting machinery is unnecessary. Our expected advantage lies elsewhere. On clinical NLI, a calibrated member can be confidently wrong on exactly the cases where heavier members disagree; the 28% faithfulness reported by iML suggests such cases are common. Agreement between two cheap members can be more informative than one member's confidence. The matched-compute comparison in the Baselines section settles this question. If the cascade wins, we would report that weighted multi-model stopping offers no gain over cascading in this domain and drop the claim.

Why not just run BEEM on the single best domain model? BEEM already turns internal exits into weighted voters, and its speedups are substantial. However, its ensemble is limited to one model's correlated layers. On NLI4CT, the gap between the best open model and both the full ensemble and GPT-4 suggests that heterogeneous members fix errors a single backbone cannot. The comparison against single-model BEEM shows whether heterogeneity is worth its memory cost.

Why not just merge the specialists into one checkpoint? Merging removes inference overhead. However, MergeBench reports that merged small models recover only about 80% of their specialist performance, and that tuning the merge costs 5 to 10 times the merge itself. A merged model also cannot vary compute per input. We would include a TIES-merged checkpoint of the size-matched members as a zero-overhead reference. This is only possible where the members' architectures allow merging.

Decisive Checks

Three results would decide the proposal. The first is whether the learned flip rule dominates the confidence cascade across the compute–accuracy curve on NLI4CT and PubMedQA. The second is whether early stopping preserves NLI4CT consistency and faithfulness relative to the full ensemble, or instead quietly trades them away for speed. The third is whether domain-pretrained members reach a stable provisional vote at shallower depth than their size-matched general counterparts. Fit-on-practice versus test-split fidelity would show how much the CeeBERT-style distribution-shift concern bites. A large gap would point to bandit-style online adaptation of $\epsilon$ as the next step.