Ensembles made of networks with different architectures are better calibrated than ensembles of one architecture trained from several random seeds. Being well calibrated means their stated confidence matches how often they are right. Building such ensembles is expensive, though. Neural Ensemble Search, which searches over architectures to assemble an ensemble, trains hundreds of candidate networks to completion and then greedily picks a few that work well together (Neural Ensemble Search for Uncertainty Estimation and Dataset Shift). The obvious way to cut that cost is to train candidates only briefly, choose the ensemble from those brief runs, and fully train only the chosen few. We argue this shortcut fails for a specific, fixable reason. The usual selection rule picks the subset with the lowest ensemble log-loss on validation data. That rule cannot distinguish a candidate that adds new information from a candidate that merely corrects the ensemble's overall confidence level. Briefly trained networks differ wildly in confidence level from one architecture to the next, so this confusion is at its worst exactly when training is cheap.
The proposal is to remove the confusion before selecting. Each candidate first gets its own fitted temperature. Temperature scaling divides a network's raw output scores by one learned number, which sharpens or softens its confidence without changing its predictions. Subsets are then scored by the log-loss they would have after the same calibration step that will be used at deployment. To keep the selection honest, temperatures are fit on one half of the validation data and subsets are scored on the other half. This is the out-of-fold discipline that stacked generalization (training a second-level model on held-out predictions of base models) uses to avoid leakage. The claim is that, with this normalization, ensembles chosen from short training runs match the ensembles that full-pool Neural Ensemble Search would choose, at a fraction of the pool-building compute.
Core Claim
Greedy ensemble selection by validation loss conflates two things: complementarity, meaning a member is right where the others are wrong, and confidence correction, meaning a member's sharpness pulls an over- or under-confident ensemble toward the right scale. Only the first survives once the final ensemble is temperature-scaled, and the work on joint calibration shows the final ensemble should be temperature-scaled (On Joint Regularization and Calibration in Deep Ensembles). Selecting on uncalibrated loss therefore optimizes the wrong target. The mismatch grows when candidates are trained for few epochs, because architectures that learn at different speeds end up at very different confidence scales.
We define the selection score of a candidate subset $S$ as
$$ \mathrm{score}(S) = \min_{T>0} \; \mathrm{NLL}_{B}\Big(\mathrm{temp}_T\big(\tfrac{1}{|S|}\textstyle\sum_{m\in S}\mathrm{softmax}(z_m / T_m)\big)\Big). $$
Here $z_m$ are candidate $m$'s output scores and $T_m$ is its individual temperature, fit on validation half $A$. The operation $\mathrm{temp}_T$ rescales the log of the averaged probabilities by a shared temperature $T$, and $\mathrm{NLL}_B$ is negative log-likelihood measured on validation half $B$. Our prediction is that the subset ranking this score produces after a short training budget agrees closely with the ranking produced after full training. Ordinary ensemble NLL, by contrast, should agree poorly in exactly the cases where candidates' confidence scales diverge most.
Relation to Prior Work
Neural Ensemble Search (Neural Ensemble Search for Performant and Calibrated Predictions; Neural Ensemble Search for Uncertainty Estimation and Dataset Shift) supplies the method we modify and the evidence that motivates it. It builds a pool by random search or evolution, selects members by forward greedy selection on raw ensemble loss, and averages them. It names pool building as its bottleneck and never applies temperature scaling inside selection. We keep its pool and greedy search and change only the scoring and the training budget of the pool.
Two papers constrain the design. Should Ensemble Members Be Calibrated? proves that calibrating each member does not yield a calibrated ensemble, so per-member temperatures alone are not enough. This is why the score also applies a shared temperature after averaging. On Joint Regularization and Calibration in Deep Ensembles shows that tuning each member separately leaves an "ensemble optimality gap," but it studies only single-architecture ensembles and does no subset selection. We carry its joint-tuning lesson into the selection step itself.
Stacking enters as a warning and as a procedure. Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap finds that a logistic-regression stacker gives the worst calibration among ensemble methods. In the same study, greedy selection was the only ensemble method that approached the calibration of the best single model. This is why our combiner is restricted to temperatures and uniform averaging rather than a learned second-level model. We borrow only stacking's cross-fitting idea.
Two lines of cheaper search are alternatives rather than ingredients. One is weight-sharing supernets for ensembles: One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking reports ImageNet results within 12 GPU days but measures no calibration. The other is zero-cost proxies (Zero-Cost Proxies for Lightweight NAS; EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring), which score a network at initialization but produce no meaningful predictions, so they cannot measure complementarity.
Research Program
We sample 500 architectures from the NAS-Bench-201 cell space (NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search) and train each on CIFAR-10, CIFAR-100 and ImageNet-16-120. We save per-example validation and test logits at several budgets, roughly 6, 12, 25, 50 and 200 epochs. We must train these networks ourselves because the benchmark's lookup tables store aggregate metrics, not per-example predictions. This one-time cost yields a reusable resource: for any budget, we can run selection on cheap logits and then read off how the same subset performs at 200 epochs.
The analysis has three parts. First, we measure how much candidates' optimal temperatures spread across architectures at each budget. Second, we measure how often uncalibrated greedy selection adds a member that lowers raw ensemble loss but not temperature-scaled ensemble loss. We call such a member a "confidence-corrector." Third, we measure rank agreement between low-budget and full-budget subset scores under each scoring rule. The method itself selects $K$ members (with $K$ between 3 and 10) from the low-budget pool and fully trains only those. Its cost is reported in GPU-hours of total training, so pool size, budget and $K$ trade off explicitly.
Baselines
The naive reference point is the single best architecture in the sampled pool, trained fully and temperature-scaled. For context, the best NAS-Bench-201 architectures reach 94.37% on CIFAR-10, 73.51% on CIFAR-100 and 47.31% on ImageNet-16-120 as reported by NAS-Bench-201. The no-search ensemble reference is a deep ensemble of $K$ seeds of that architecture with one joint temperature, following On Joint Regularization and Calibration in Deep Ensembles. We give it the same total training compute that our method spends.
The strongest existing method is full-pool Neural Ensemble Search: all 500 candidates trained for 200 epochs, forward selection on raw ensemble NLL, uniform averaging, and a post-hoc joint temperature added so that the comparison concerns selection rather than final calibration. On CIFAR-10, Neural Ensemble Search for Uncertainty Estimation and Dataset Shift reports up to 40% lower ECE than deep ensembles, and a 6.6x speedup of its evolutionary variant over random search on NAS-Bench-201. The file gives no per-dataset ensemble NLL or error values, so we reproduce this baseline in our own pool. Beating it means matching its test NLL and ECE within seed noise at 15% or less of its pool-building compute, or beating its NLL at equal compute.
The key ablation runs the same low-budget pipeline with raw ensemble NLL as the selection score. This isolates temperature normalization. A further ablation drops cross-fitting and fits temperatures and selects on the same validation data, to isolate validation overfitting. A logistic-regression stacker trained out-of-fold on the selected members tests whether the calibration trap reported for tabular models reappears here.
Why not just…?
Why not just use a deep ensemble of the best architecture with joint temperature scaling? It is cheap and well understood. Neural Ensemble Search, however, found that seeds of one architecture share failure modes, and that heterogeneous ensembles cut ECE by up to 40% on CIFAR-10 with gains that widen under corruption shift. Our compute-matched comparison on CIFAR-10-C will show whether that advantage survives once the deep ensemble is also jointly temperature-scaled. If it does not survive, the motivation for any ensemble search shrinks, and we will report that.
Why not just run greedy selection on the short-training logits without temperatures? This is the real threat. If confidence scales at low budget turn out to be similar across architectures, the ablation will match our method and the contribution reduces to an observation. The temperature-spread and confidence-corrector measurements decide this directly, and we treat agreement between the two pipelines as a publishable negative result.
Why not just keep the top candidates by short-training accuracy, train them fully, then run ordinary selection? This successive-halving style filter ranks models individually, and the ensemble-pruning literature argues that complementary members are often not the individually best ones (Hierarchical Pruning of Deep Ensembles with Focal Diversity). We run it at matched compute. If it ties our method, then complementarity is not what the cheap stage needs to capture.
Why not just use zero-cost proxies or a weight-sharing supernet? Zero-cost proxies give one number per network at initialization and no predictions, so they cannot score how well members combine. We include them as a pre-filter baseline. One-shot ensemble search reports no calibration (One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking), and shared-weight rankings are known to diverge from standalone training. Evaluating a supernet-derived ensemble under our calibration metrics is a useful secondary comparison but would not settle the selection question.
Proposed Experiments
The decisive check is a curve on each dataset. It plots fully trained test NLL and ECE of the selected ensemble against the training budget used for selection, comparing our score with raw-NLL selection and with the full-pool Neural Ensemble Search reference. The hypothesis predicts two things. Our curve should flatten near the reference at budgets of 12 to 25 epochs. The raw-NLL curve should stay above it, and the gap should be largest where the spread of fitted temperatures is largest. Spearman agreement between low-budget and full-budget subset scores is the mechanistic check. CIFAR-10-C extends the test to dataset shift, where a temperature fit on clean validation data is known to transfer imperfectly (Robust Calibration with Multi-domain Temperature Scaling).
As a reference for robustness, UraeNAS reports gains over DrNAS within the NAS-Bench-201 space of "0.86% in accuracy, 42% in ECE" on CIFAR-10 and "2.43% in accuracy, 30% in ECE" on CIFAR-10-C (Unified Probabilistic Neural Architecture and Weight Ensembling Improves Model Robustness). We will compare against those figures where our protocol allows. The claim is falsified if raw-NLL selection matches temperature-normalized selection at every budget, or if neither low-budget pipeline approaches the full-pool reference before 50 epochs.
References
- Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
- On Joint Regularization and Calibration in Deep Ensembles
- Neural Ensemble Search for Performant and Calibrated Predictions
- Should Ensemble Members Be Calibrated?
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
- One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking
- Zero-Cost Proxies for Lightweight NAS
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Hierarchical Pruning of Deep Ensembles with Focal Diversity
- Robust Calibration with Multi-domain Temperature Scaling
- Unified Probabilistic Neural Architecture and Weight Ensembling Improves Model Robustness