Routing Ensembles by Imitating Their Own Disagreement

A deep ensemble, meaning several independently trained networks whose predictions are averaged, is still the most reliable cheap way to get calibrated probabilities and to notice inputs the model has never seen. The members disagree on those inputs, and that disagreement is what you pay for. You pay $N$ times the compute on every input, including the large majority where all members agree. One study found that over 77% of CIFAR-100 test images are already classified correctly by a single ResNet-18 (Towards Inference Efficient Deep Ensemble Learning). The obvious fix is a gate: a small network that looks at an input and decides which members to run and how to weight them. Gates of this kind are standard in adaptive feature fusion, where they replace fixed addition or concatenation with input-dependent weights. The risk is that a gate trained to minimize classification loss learns to favor whichever member is currently most accurate. That is the same collapse that makes jointly trained ensembles fail. It also quietly deletes the disagreement signal that justified the ensemble in the first place.

We propose training the gate to reproduce the full ensemble's output distribution instead of to predict labels. The gate runs members one at a time, in an order it picks per input rather than a fixed sequence. After each step it reads how much the members already executed disagree in their intermediate feature maps. It stops once the weighted mixture of executed members is predicted to match what all $N$ members would have said. On inputs where the full ensemble would disagree, which are exactly the hard and out-of-distribution ones, the gate is pushed to keep spending compute. The claim to test is that this target, not the gating architecture, is what lets an adaptive ensemble cut compute without giving back the calibration and out-of-distribution detection that ensembles exist to provide.

Core Claim

Existing compute-saving ensembles optimize accuracy per unit of compute. IRENE trains a recurrent selector with an accuracy loss, an efficiency penalty and a ranking loss over a fixed member order, and reaches 95.81% on CIFAR-10 versus 95.62% for the full 10-member ResNet-18 ensemble at up to 56% lower average cost (Towards Inference Efficient Deep Ensemble Learning). That paper does not report what happens to calibration or out-of-distribution detection. An accuracy-driven halting rule has a clear failure mode: it stops early on any input where the first member is confident. Single networks are confidently wrong precisely on shifted inputs, so those are the inputs where it will stop too soon.

Our selector is trained against a different target. Let $p_k(x)$ be member $k$'s class distribution and $\bar p(x) = \frac{1}{N}\sum_{k=1}^N p_k(x)$ the full uniform ensemble. For an executed subset $S(x)$ with convex weights $w_k(x)$ produced by the gate, we minimize

$$ \mathcal{L} = \mathbb{E}_x\Big[\mathrm{KL}\big(\bar p(x)\,\big\|\,\textstyle\sum_{k\in S(x)} w_k(x)\,p_k(x)\big)\Big] + \beta\,\mathbb{E}_x\big[|S(x)|\big], $$

where $\beta$ sets the price of running one more member. Because the target $\bar p$ already contains the members' disagreement, the gate gains nothing from collapsing onto one member. It is rewarded only for spending compute where stopping would change the ensemble's answer or its spread. Two consequences follow, and both are tested below. First, the target needs no labels, so the gate can be trained on unlabeled and even shifted data. Second, the claim is falsifiable: if a plain confidence threshold on the first member already matches full-ensemble behavior at equal compute, the learned gate is pointless.

Relation to Prior Work

Two lines of work motivate the proposal by exposing the gap. To Ensemble or Not Ensemble: When Does End-to-End Training Fail? shows that training ensemble members under one joint loss produces "model dominance" in over-parameterized networks. One member wins early, the others are starved of gradient, and the optimization becomes ill-conditioned. That paper keeps aggregation static. We read its result as a warning that a task-loss-trained gate will reproduce the same dominance at the routing level, and we test this directly as an ablation. Coopetitive Soft Gating Ensemble weights base models by global accuracy and by local k-nearest-neighbor accuracy, interpolating between averaging and hard selection. It improves Boston Housing RMSE from 23.2 to 18.9. However, it runs every member and adds a neighbor search, so it raises cost rather than lowering it, and it is evaluated only on classical tabular models.

Three papers supply the mechanism. From Attentional Feature Fusion we take the multi-scale channel attention block. It combines a globally pooled branch with an unpooled local branch, and its authors show it beats a deeper ResNet-101 on ImageNet with fewer parameters (20.2% vs 23.2% top-1 error). We use it as the module that reads intermediate feature maps of executed members and summarizes where they agree. From Lightweight Attentional Feature Fusion we take the softmax convex-combination head. On MSR-VTT it reached 0.358 mAP against 0.306 for multi-head self-attention and 0.247 for concatenation, at 27.8M versus 94.9M FLOPs. Those results argue that a cheap convex weighting is sufficient and that heavier attention is unnecessary. From Breaking the Echo Chamber we take the observation that routers trained greedily make experts redundant, and that a diversity-aware routing objective can be added at about 3% training overhead and no inference cost. Our distillation-to-ensemble target is a different answer to the same collapse, applied to calibration and uncertainty, which that paper does not measure.

One paper constrains the proposal and may undercut it. Improving Accuracy and Calibration via Differentiated Deep Mutual Learning trains peer networks and discards them at test time. It reaches 79.05% CIFAR-100 accuracy with ResNet-50 and ECE of 0.86%, down from 10.04%, at single-model inference cost, and it matches a 3-member deep ensemble on SVHN out-of-distribution detection. If a zero-overhead single model already matches a small ensemble on the uncertainty metrics, an adaptive ensemble has to earn its extra compute somewhere else. We treat that as the central threat.

Parameter-virtualized ensembles such as Single Model Ensemble using Pseudo-Tags and Distinct Vectors save memory but still require $K$ full passes. Our selector is agnostic to how members are stored, so it could cut those passes too. We leave that as a follow-up rather than the core test.

Research Program

The members are $N=10$ independently trained ResNet-18s on CIFAR-10 and CIFAR-100, matching IRENE's setting so its numbers are directly comparable. A second configuration uses five ResNet-50s on CIFAR-100, matching Diff-DML's backbone. Members are frozen and only the selector is trained, which keeps the comparison about routing rather than about better members.

At step $t$ the selector holds the executed set $S_t$. It pools each executed member's feature maps after stages 2 and 4 and passes them through a multi-scale channel attention block. It then computes cross-member agreement statistics: the cosine similarity of the attended features and the spread of predicted class distributions. From these it emits three outputs: a stop probability, a distribution over which unexecuted member to run next, and convex fusion weights over $S_t$. The first member is chosen by a learned prior.

Training uses a held-out unlabeled split and the objective above. The discrete choices are handled with a straight-through Gumbel relaxation. Sweeping $\beta$ traces a compute curve.

The analysis has three parts. First, we measure whether the gate's continuation decisions correlate with the full ensemble's mutual information, a standard measure of member disagreement, on in-distribution versus out-of-distribution inputs. Second, we track whether learned fusion weights stay near uniform or drift toward one member. Third, we check whether training the gate on a mix of clean and unlabeled shifted images, which the label-free target permits, changes behavior on shifted test data.

Baselines

The naive reference points are a single member and the full uniform 10-member ensemble. Every method is plotted between them on curves of accuracy, ECE and out-of-distribution AUROC against average members executed. A random subset of $m$ members averaged uniformly is added at each $m$, to show what adaptivity buys over just running fewer models.

The strongest existing method on this question is IRENE (Towards Inference Efficient Deep Ensemble Learning). It is the only cited method that adaptively decides how many deep ensemble members to run, and it reports 95.81% CIFAR-10 accuracy versus 95.62% for the full ensemble at up to 56% lower cost with ResNet-18. We reimplement it on our frozen members, with its recurrent selector and ranking loss, and give it the same held-out split and the same hyperparameter search budget as our selector. Our method beats it if, at matched average compute, it matches IRENE's accuracy and achieves lower ECE and higher SVHN out-of-distribution AUROC, with the gap concentrated on SVHN inputs. IRENE did not report calibration, so there is no reported calibration number for IRENE on Lacuna, and we will measure it.

The simplest strong competitor is a confidence cascade. It runs members in a fixed order and stops when the running average's maximum softmax probability exceeds a threshold tuned on the same held-out split. I found no reported result on Lacuna for a confidence-threshold cascade on deep ensembles. It needs no training at all, so it is the bar any learned gate must clear.

The zero-overhead competitor is Diff-DML (Improving Accuracy and Calibration via Differentiated Deep Mutual Learning). Its reported CIFAR-100 ResNet-50 numbers are 79.05% accuracy and 0.86% ECE. We train it from its published recipe on the same data and report it as a single point on every curve. Temperature scaling of one member is reported alongside it.

Two ablations isolate the key ingredient. The first trains the identical selector with cross-entropy on labels plus the same cost penalty, in place of the ensemble-imitation target. The second keeps our target but forces uniform weights over executed members, separating the routing contribution from the weighting contribution. A third ablation replaces the feature-agreement inputs with output-probability inputs only, to test whether looking inside the members matters.

Why not just…?

Why not just threshold the first member's confidence? This is the most serious threat on clean data. IRENE's 77% observation suggests most inputs stop after one member under any sensible rule. The weakness should appear only on shifted inputs, where a single member is confident and wrong and a confidence rule stops exactly when it should not. The experiment that settles this is out-of-distribution AUROC and ECE on SVHN at matched compute. If the cascade matches us there too, the learned selector adds nothing and we will say so.

Why not just distill the ensemble into one network, as Diff-DML does? Diff-DML gives single-model cost and reported strong calibration, and it matched a 3-member ensemble on SVHN detection. It cannot, however, adapt its compute to an input, and its uncertainty comes from one network's softmax rather than from measured disagreement. Our method justifies itself only if, at an average cost slightly above one member, it beats Diff-DML on out-of-distribution AUROC or on ECE under shift. If Diff-DML dominates our curve everywhere, distillation is the better answer and the adaptive-ensemble program should stop.

Why not just gate with the task loss, as most adaptive fusion modules do? Fusion gates like those in Offline Reinforcement Learning with Adaptive Feature Fusion and Attentional Feature Fusion are trained end to end on the task objective, and that works for accuracy. The model-dominance results in To Ensemble or Not Ensemble predict that the same objective will push a routing gate toward the lowest-loss member and away from diversity. The label-trained ablation measures this directly through the entropy of the fusion weights and the gate's correlation with ensemble mutual information.

Decisive Checks

The proposal succeeds only if three results hold together. First, on CIFAR-10 at roughly IRENE's cost level, the selector stays within IRENE's accuracy and within a small margin of full-ensemble ECE and SVHN AUROC. Second, the label-trained ablation shows measurably more weight collapse and worse out-of-distribution AUROC at the same compute. Third, the gate executes more members on SVHN inputs than on CIFAR test inputs, which is the behavioral signature that it learned to track disagreement rather than difficulty alone.

A clean negative result is also informative and publishable. If the confidence cascade or Diff-DML matches the selector on shifted data, the field would know that adaptive ensemble routing buys accuracy per FLOP but not uncertainty per FLOP. No paper in this file has measured that yet.