Running a whole ensemble (several independently trained models whose predictions are averaged) on every input is expensive. Most of that compute is wasted on easy inputs, so a common fix is to evaluate members one at a time and stop once the answer looks settled. Existing stopping rules have a blind spot, though. They keep recruiting members whenever the prediction is uncertain, and they cannot tell whether the uncertainty exists because the models disagree or because the input itself is ambiguous: a blurry image, a mislabeled example, or a genuinely two-way case. Extra members help only in the first situation. In the second, every new model repeats the same hedged 50/50 answer, and the budget drains into what the cascade literature calls a "noise sink."
We propose a sequential ensemble that decides how many members to run by splitting uncertainty into two parts after the first two members. The first part is disagreement between members (epistemic uncertainty, meaning the models lack knowledge a further member might supply). The second is uncertainty each member reports on its own (aleatoric uncertainty, meaning noise in the data that no model can remove). The rule has two parts. It recruits more members only when disagreement is high. It stops early both when members confidently agree and when they agree the input is ambiguous. The claim is concrete and falsifiable: at equal accuracy, this rule should use substantially less compute than existing confidence-driven early-exit ensembles, and the savings should concentrate on inputs that humans also find ambiguous.
Core Claim
The closest existing method is Adaptive Ensemble Prediction for Deep Neural Networks based on Confidence Level. It samples ensemble predictions one by one and stops when a Student's-t confidence interval on the top class no longer overlaps the intervals of competing classes. On ImageNet it reports a 12.7x speedup over a 20-prediction ensemble at nearly identical accuracy, with average evaluations falling from 10–20 to 1.5–6.
That rule has a specific failure. On an input where every member outputs roughly 0.5/0.5 over two classes, the intervals narrow around two overlapping means, never separate, and the rule runs to the maximum budget. It spends the most compute exactly where ensembling cannot help. Inoue's own analysis points the same way: ensembling pays off near decision boundaries, not on errors a model makes with high confidence. It does not, however, separate boundary cases caused by model ignorance from boundary cases caused by label ambiguity.
When Does Confidence-Based Cascade Deferral Suffice? names this noise-sink failure for two-model cascades. It shows that the optimal deferral rule depends on the gain in correctness from the next model, not on the current model's confidence. Its fix is a small learned network (called Diff-01) that reads the first model's outputs and predicts that gain.
Our observation is that a single model's output cannot distinguish the two kinds of uncertainty, while two members' outputs can. The classical decomposition makes this precise. Let $p_1,\dots,p_k$ be the class-probability vectors of the first $k$ members, $\bar p$ their average, and $H$ the entropy function. Then
$$H(\bar p) = \frac{1}{k}\sum_{j=1}^{k} H(p_j) + \mathrm{MI}_k,$$
where the first term is the average member entropy (aleatoric) and $\mathrm{MI}_k$ is the mutual information between the prediction and member identity, which measures disagreement (epistemic). The proposed rule after member $k \ge 2$ has three outcomes. It exits if $\max \bar p$ is high. It also exits if $\mathrm{MI}_k$ is below a threshold $\tau_e$, even when $H(\bar p)$ is large; this is the "agreed ambiguity" case. It recruits member $k+1$ otherwise.
The thresholds are not raw values. They are set on calibrated components, using the two-parameter post-hoc scaling from JUCAL. That method rescales member outputs for data noise ($c_1$) and member disagreement ($c_2$) separately, and it reports NLL reductions of up to 15%. JUCAL was designed only to calibrate a fixed ensemble. We reuse its calibration so that the two uncertainty terms are on comparable scales before they drive a stopping decision.
Research Program
The system has three stages. Member 1, the cheapest, always runs, and a standard confidence exit handles clearly easy inputs, as in On the Surprising Efficiency of Committee-based Models. That paper reports a 5.4x FLOP reduction versus EfficientNet-B7 on ImageNet using plain maximum-probability exits. Inputs that survive this exit get member 2. From then on, the uncertainty decomposition governs whether to continue. Members come from a pool of independently trained deep ensemble members, including cross-architecture members, in a fixed cheapest-first order. Learned ordering is deliberately left out, so that the stopping rule is the only new ingredient.
The analysis has three parts, and each tests a mechanism rather than only reporting accuracy per FLOP.
The first part tests whether low disagreement combined with high member entropy actually identifies inputs that further members cannot fix. We measure, per input, how much accuracy the full ensemble adds over the two-member ensemble, split by quadrant of (aleatoric, epistemic).
The second part tests whether the rule depends on real functional diversity. Is BatchEnsemble a Single Model? shows that BatchEnsemble members have near-zero disagreement and near-identical weights. Its CIFAR-10C calibration error (~0.308) is close to a single model's (~0.323), against ~0.206 for deep ensembles. With such members, our $\mathrm{MI}_k$ would be near zero everywhere, and the rule would wrongly stop on everything uncertain. This is a constraint on the design, and we test it directly by swapping the pool.
The third part tests whether error correlation across architectures limits the idea. Human and AI Perceptual Differences in Image Classification Errors reports that pairing two strong models yields only a ~7% oracle boost on CIFAR-10, versus +10.86% for pairing a model with a human. If members fail on the same inputs, low disagreement can mean a shared blind spot rather than real ambiguity. We therefore report how often "agreed ambiguity" exits turn out to be shared errors on cleanly labeled images.
Baselines
The naive reference points are a single member and a full static ensemble with uniform averaging, both run on the same pool. On CIFAR-10, Is BatchEnsemble a Single Model? reports 94.1% for a single model and 95.2% for deep ensembles. These bracket the accuracy range and fix the full-cost endpoint. Beating them means reaching within 0.1 points of full-ensemble accuracy at a fraction of its FLOPs.
The strongest direct competitor is Inoue's confidence-interval early exit, run on the identical member pool, member order, and maximum budget, with its confidence level swept to trace a full accuracy-versus-average-members curve. Its reported ImageNet result is a 12.7x speedup at nearly identical accuracy, with an average of 1.5–6 evaluations. Note that this result used test-time crop augmentation rather than independent members, so we rerun it on our pool rather than compare against the number directly. Our method beats it if its curve dominates Inoue's across the sweep, and specifically if it uses fewer members on high-entropy inputs at matched accuracy.
The strongest learned competitor is a Diff-01-style gate following When Does Confidence-Based Cascade Deferral Suffice?: a three-layer network that reads member-1 features (entropy, top-10 probabilities) and predicts whether the full ensemble would add correctness. The file reports no exact number for Diff-01 (no reported result found on Lacuna for its accuracy on ImageNet). We train it on the same held-out split and tuning budget as our thresholds.
The ablation that isolates the key ingredient is a version that also runs two members but gates on total entropy $H(\bar p)$ instead of $\mathrm{MI}_k$. If this matches the full method, the decomposition is doing nothing. A second ablation drops the JUCAL calibration and uses raw components.
Why not just…?
Why not just use Inoue's confidence-interval rule with a lower maximum budget? A lower cap limits how much a noise sink can cost, but it limits the cost everywhere. That includes shifted or boundary inputs where more members genuinely help. The cap trades accuracy on epistemic cases for savings on aleatoric ones, while our rule aims to keep both. The swept comparison in the CIFAR-10H experiment below settles this: if a capped Inoue curve matches ours, the idea adds nothing.
Why not just train a learned gate on the first model, as Diff-01 does? This is the real threat. Our argument is informational: one model's output cannot show whether a second model would disagree, whereas two members' outputs reveal that directly. A strong gate might still learn to predict ambiguity from member-1 features alone, for example if ambiguous images have recognizable signatures. We give the Diff-01 gate the same data, and we also test a variant that sees both members' outputs. If the two-member learned gate beats our threshold rule, we report the decomposition as the right features, not the right rule, and adopt the learned version.
Why not just run a small calibrated ensemble on everything? JUCAL reports that a calibrated 5-member ensemble can match 50 uncalibrated members on calibration. A fixed small committee is therefore a credible compute-light option. We run a JUCAL-calibrated fixed ensemble of each size from 2 to 5, and our method must dominate these points on accuracy and NLL per FLOP. The mechanism predicts it will, because most inputs should exit after one member.
Proposed Experiments
The decisive benchmark is CIFAR-10 with the CIFAR-10H soft labels and CIFAR-10N noisy labels described in Human and AI Perceptual Differences in Image Classification Errors. Human label entropy gives an independent, per-image measure of real ambiguity, so we can check whether the router's "agreed ambiguity" exits match images humans also disagree on. The central prediction is that, among images in the top decile of human label entropy, our method uses at least 30% fewer members than Inoue's rule at matched overall accuracy. The same should not hold for the bottom decile. If the savings are spread evenly across human-entropy deciles, the mechanism claim is false even if the aggregate curve looks good.
CIFAR-10C tests the opposite regime, distribution shift, where uncertainty should be mostly epistemic. Here the method should recruit more members than on clean data and recover most of the deep-ensemble calibration gain (~0.206 ECE versus ~0.323 single-model, from Is BatchEnsemble a Single Model?). CIFAR-100 and ImageNet validation provide the scale check against the published compute-saving results of Inoue and of Learning to Cascade, which reports a 30% MAC reduction for a ResNet152 cascade on ImageNet. Every configuration reports accuracy, NLL, ECE, average members evaluated, and FLOPs, over five seeds of pool training.
Two checks determine whether the idea survives. The first is the total-entropy ablation: if gating on $H(\bar p)$ after two members performs as well as gating on $\mathrm{MI}_k$, the decomposition is not the operative ingredient. The second is the BatchEnsemble pool swap: the method should degrade sharply with low-diversity members, showing that it depends on real disagreement, as the analysis predicts.
References
- Adaptive Ensemble Prediction for Deep Neural Networks based on Confidence Level
- When Does Confidence-Based Cascade Deferral Suffice?
- JUCAL: Jointly Calibrating Aleatoric and Epistemic Uncertainty in Classification Tasks
- On the Surprising Efficiency of Committee-based Models
- Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles
- Human and AI Perceptual Differences in Image Classification Errors
- Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems