An ensemble of neural networks, meaning several independently trained models whose predictions are averaged, is still the most reliable way to get honest confidence estimates when test data drift away from training data. It costs M times as much to run as one model. The cheap substitutes pack all members into one network. BatchEnsemble gives each member a rank-one tweak of shared weights, and MIMO trains several subnetworks inside one wide backbone. Both copy one hand-designed backbone M times, and the evidence suggests this homogeneity is where diversity dies. Separately, neural ensemble search showed that members with different architectures make more different errors than same-architecture members with different random seeds. That work used fully separate networks, so it paid the full M× inference bill.
This note proposes a single-pass ensemble made of one shared early-layer "stem" followed by M branches. Each branch is a different, automatically searched cell topology. During training, each branch sees stem features of a different input example, which keeps the branches' errors independent. At test time, the stem runs once and its output feeds every branch. The question is whether topological heterogeneity, chosen by search under a fixed compute budget, closes the calibration gap between cheap ensembles and full deep ensembles. The design is also meant to separate three possible sources of any gain: the topology, the search, and the training trick.
Core Claim
At a fixed inference cost of roughly one to one-and-a-half single models, a network built from searched heterogeneous branches will reach uncertainty quality under corruption and on out-of-distribution data that cheap homogeneous ensembles do not reach. Uncertainty quality here means negative log-likelihood (NLL), expected calibration error (ECE), and out-of-distribution detection AUROC. Two ingredients are needed together. The first is architectural mismatch between branches. The second is a training scheme that stops a shared stem from pulling the branches back into the same solution. The proposal has two kinds of result. One is the method. The other is a decomposition of where efficient-ensemble diversity actually comes from, and the decomposition stays useful even if the method only partly succeeds.
Where This Sits
Two papers motivate the gap. Is BatchEnsemble a Single Model? shows that rank-one members of BatchEnsemble end up with weight cosine similarity near 1.0 and near-zero disagreement. On CIFAR-10 versus SVHN, BatchEnsemble reaches AUROC 0.888, below a single model's 0.893. Training independent subnetworks for robust prediction, which introduced MIMO, gets real independence by feeding each subnetwork a different input during training. Its capacity saturates at about two to four members, because all members share one homogeneous backbone.
Two further papers supply the hypothesis and a design element. Neural Ensemble Search (NES) shows that architecture diversity places members in distinct regions of function space and reduces ECE by up to 40% relative to deep ensembles. It builds its ensembles from disjoint full networks, so it does nothing about inference cost. NEAS is the closest prior work. It searches a shared stem, a split point, and two heterogeneous branches under a FLOP budget. But it optimizes only ImageNet top-1 accuracy, averages two branches uniformly, never measures calibration or behaviour under distribution shift, and trains the branches on identical inputs.
We differ from NEAS in three ways. We search for ensemble NLL under shift rather than accuracy. We allow M up to six. We decorrelate branch training with per-branch input permutation, borrowed from MIMO. We then test whether each of these choices matters.
One paper constrains the search method. NAS-Bench-201 documents that differentiable search collapses onto parameter-free skip connections and loses to random search and regularized evolution. We therefore use regularized evolution and random search, not DARTS-style relaxation.
Research Program
The network keeps the macro skeleton of NAS-Bench-201: a shared stem of searchable depth, followed by M parallel branches. Each branch stacks one cell from the NAS-Bench-201 cell space, so its topology can be looked up and is reproducible. Each branch ends in its own classifier head.
Training borrows MIMO's independence trick and moves it past the stem. For a mini-batch of size B, the stem processes the batch once. Branch i then receives the stem features in a branch-specific random permutation, and its loss is computed against the correspondingly permuted labels. The stem receives gradients from all branches, but each branch fits a different pairing of examples at every step, which decorrelates branch errors. Because the stem runs once per batch, training costs about the same as standard training of the full network. At test time there is no permutation: one stem pass feeds all branches, and the branch predictions are averaged.
The search space has three parts: the split depth, the M cell topologies, and M itself (two to six). Branch widths are set so that total multiply-accumulate operations (MACs) stay within a target budget. The search objective is ensemble NLL on a held-out subset of CIFAR-10-C corruption types. This follows the shifted-validation selection used in NES. Test corruption types are kept disjoint from the corruption types used for selection, so the search cannot overfit to the evaluation shift. Each candidate is trained for a short proxy schedule of 30 epochs. We will report the rank correlation between the proxy scores and full-training scores on a sample of candidates, so readers can see how far the proxy can be trusted.
The analysis compares searched, randomly drawn heterogeneous, and homogeneous branch sets. For each, it measures pairwise disagreement between branches, weight and feature similarity, and NLL under shift. It also measures how quickly ensemble gains saturate as M grows, to test whether heterogeneity pushes back MIMO's ceiling of about four members.
Baselines
The naive reference point is a single network at the same MAC budget: the best NAS-Bench-201 architecture widened to match. NAS-Bench-201 reports its best cell at 94.37% on CIFAR-10. Is BatchEnsemble a Single Model? reports a single ResNet baseline at ECE about 0.323 on CIFAR-10-C. Both numbers come from different backbones and are not our matched single-model baseline. We retrain the widened NAS-Bench-201 cell ourselves under the same training recipe.
The upper reference is a four-member deep ensemble, which costs M× inference. The cited paper reports it at 95.2% accuracy, ECE about 0.206 on CIFAR-10-C, and AUROC 0.924 on SVHN. Closing at least half of the CIFAR-10-C ECE gap between our single-model baseline and this ensemble, at roughly one-quarter of its cost, would count as a clear success.
The strongest efficient competitor is MIMO. Its paper reports, for M=3 on CIFAR-100, 82.0% accuracy and NLL 0.690, against 79.8% and 0.875 for a deterministic model. We reimplement MIMO on the same widened NAS-Bench-201 backbone at matched MACs, rather than relying on its WideResNet numbers. Beating it means lower NLL on CIFAR-100-C and higher SVHN AUROC across three seeds.
We also run three further baselines at matched MACs and an equal hyperparameter tuning budget per method. The first is BatchEnsemble, whose original paper reports 80.32% on CIFAR-100. The second is a NEAS-style variant: same space, accuracy objective, two branches, no input permutation. The third is NES-RS with members shrunk to fit the same total budget.
Three ablations isolate the key ingredients. The first replicates the single best cell across all branches, which keeps the training scheme and removes heterogeneity. The second uses randomly drawn heterogeneous cells, which keeps heterogeneity and removes the search. The third removes the per-branch permutation, which keeps topology and removes decorrelated training.
Why not just…?
Why not just use MIMO or BatchEnsemble on a good hand-designed backbone? The cited diversity analysis shows that BatchEnsemble on homogeneous backbones barely differs from a single model. MIMO does decorrelate its members, but it saturates at a few members. The homogeneous-branch ablation answers this directly. If replicated best cells with permuted training match the searched heterogeneous model, then topology is not the lever, and we will report that result.
Why not just ensemble small, separately chosen NES members? Under a fixed budget, M disjoint networks must each be about M times smaller and cannot share early features. NES never tested ensembles at matched total cost. The shrunk NES-RS baseline settles whether a shared stem helps or hurts.
Why not just draw heterogeneous branches at random and skip the search? This is the most serious threat. NES found that random search is competitive, and NAS-Bench-201 found that random search is a strong baseline in general. If the random-heterogeneous ablation lands within seed noise of the searched model, the contribution reduces to "heterogeneous branches plus permuted training," and search becomes optional. We would state that plainly and would still report the cheaper recipe, which would itself be useful.
Why not just use a distance-aware single model? SNGP is a single network made distance-aware through spectral normalization plus a Gaussian-process output layer. A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness reports AUROC 0.846 for it on CIFAR-100 out-of-distribution detection. The same paper notes that ensembling SNGP models helps further, so the two approaches are complementary. We will include SNGP at matched MACs, and we will add an SNGP output head on each branch as an optional extension.
Proposed Experiments
The main experiments run on CIFAR-10 and CIFAR-100 with their corruption sets, as defined in Benchmarking Neural Network Robustness. SVHN serves as out-of-distribution data. ImageNet-16-120 tests whether the conclusions carry over to a third dataset. We report accuracy, NLL, ECE, AUROC, measured latency, and MACs. Every result uses three seeds.
The decisive checks are three. First, the searched model must beat matched-MAC MIMO on NLL under corruption. Second, the gap between the searched model and the random-heterogeneous ablation must exceed seed variance. Third, removing the permutation must reduce branch disagreement toward the near-zero levels reported for BatchEnsemble. Each check has a different reading if it fails: on the first, the method fails; on the second, search is unnecessary; on the third, the training trick is not the source of independence.
References
- Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles
- BatchEnsemble: an Alternative Approach to Efficient Ensemble and Lifelong Learning
- Training independent subnetworks for robust prediction
- Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
- One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations