Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, Yee Whye Teh

TL;DR

Neural Ensemble Search (NES) introduces a novel way to build neural network ensembles by automatically searching for and combining different architectures, rather than just retraining a single, fixed architecture with different random seeds. By leveraging architectural variation as a primary source of diversity, NES significantly improves classification accuracy, uncertainty calibration, and robustness to dataset shift compared to standard Deep Ensembles.

What Problem Does This Paper Address?

Deep learning models are notoriously overconfident when faced with "dataset shift"—situations where the data seen during deployment differs from the training data (e.g., a self-driving car encountering unexpected weather conditions).

To mitigate this, researchers typically use Deep Ensembles, which involve training the exact same neural network architecture multiple times with different random initializations. While Deep Ensembles represent the state-of-the-art for uncertainty estimation, they are constrained by their fixed architecture. They only explore the diversity that random initialization can provide within a single model's landscape. This paper asks a fundamental question: Can we achieve better diversity and robustness by automatically searching for an ensemble composed of entirely different network architectures?

Methods

The authors propose Neural Ensemble Search (NES), a framework that treats ensemble construction as a two-step optimization problem:

  1. Pool Building: Generate a large pool of candidate neural network architectures trained independently on the training set. The paper introduces two ways to do this:
  2. NES-RS (Random Search): Architectures are sampled uniformly at random from a modern Neural Architecture Search (NAS) space. It is simple and embarrassingly parallelizable.
  3. NES-RE (Regularized Evolution): A more guided approach based on evolutionary algorithms, where architectures are iteratively mutated over time.
  4. Ensemble Selection: From the pool, greedily select a subset of $M$ models using a forward step-wise selection algorithm (ForwardSelect) without replacement to minimize validation loss.

To make the ensemble robust to dataset shift, the authors introduce a clever twist: they perform the ensemble selection step using a validation set that has been intentionally subjected to a disjoint set of data corruptions ($D_{val}^{shift}$).

Figure 3: Illustration of one iteration of NES-RE. Network architectures are represented as colored bars of different lengths illustrating different layers and widths. Starting with the current population, ensemble selection is applied to select parent candidates, among which one is sampled as the parent. A mutated copy of the parent is added to the population, and the oldest member is removed.

Datasets

The methods were validated across two major architecture search spaces (DARTS and NAS-Bench-201) using five popular image classification datasets:

  • Fashion-MNIST
  • CIFAR-10
  • CIFAR-100
  • Tiny ImageNet
  • ImageNet-16-120

To simulate dataset shift, the authors applied 19 different types of digital corruptions (like blur, noise, and weather effects) at varying severities, following the standard benchmark established by Hendrycks & Dietterich (2019).

Key Insights and Ideas

  • Architectural Diversity vs. Individual Strength: Standard NAS focuses on finding the single best-performing architecture. However, NES demonstrates a fascinating trade-off: an ensemble of slightly weaker, diverse architectures can easily outperform a deep ensemble composed of the single "best" architecture.
  • Function Space Exploration: Using t-SNE dimensionality reduction to visualize model predictions, the authors show that different initializations of a fixed architecture cluster closely together in function space. In contrast, varying the architecture forces the models to explore entirely different regions, yielding much higher predictive disagreement and ensemble diversity.
t-SNE visualization showing that fixed architectures cluster together while varying architectures explore more of the function space

Key Results

  • Superior Calibration and Robustness: On CIFAR-10, NES reduced the Expected Calibration Error (ECE)—a metric evaluating how well a model's confidence matches its actual accuracy—by up to 40% relative to standard Deep Ensembles under dataset shift.
  • Outperforming the "Optimal" Architecture: On the NAS-Bench-201 space (where all 15,625 possible architectures are exhaustively pre-evaluated), NES ensembles consistently outperformed a Deep Ensemble made from the absolute best standalone architecture in the search space.
Figure 1: A comparison of a deep ensem- ble with the best architecture (out of 15,625 possible architectures) and ensembles con- structed by our method NES on ImageNet- 16-120 over NAS-Bench-201. Performance metrics (smaller is better): error, negative log- likelihood (NLL) and expected calibration error (ECE). We also measure average base learner NLL and two metrics for ensemble diversity (see Section 3): oracle NLL and [1−predictive disagreement]; small means more diversity for both metrics. NES ensembles outperform the deep ensemble, despite the latter having a sig- nificantly stronger average base learner.
  • Efficiency Gains: NES-RE achieved a 6.6x speedup over the random search variant (NES-RS) on NAS-Bench-201 to reach the same level of ensemble performance.

Limitations and Future Work

The primary limitation of NES is its computational cost. Building the initial pool requires training hundreds of networks from scratch. While this process is highly parallelizable, it remains resource-intensive.

The authors attempted to accelerate pool generation using popular "weight-sharing" NAS methods (where a single massive network is trained, and sub-networks are sampled from it). However, this drastically degraded performance due to "weight interference"—the sub-networks performed poorly when evaluated or retrained independently. The authors note that developing one-shot or differentiable NAS methods specifically tailored for ensemble selection is a crucial direction for future work.

Summary updated

Abstract

Ensembles of neural networks achieve superior performance compared to stand-alone networks in terms of accuracy, uncertainty calibration and robustness to dataset shift. \emph{Deep ensembles}, a state-of-the-art method for uncertainty estimation, only ensemble random initializations of a \emph{fixed} architecture. Instead, we propose two methods for automatically constructing ensembles with \emph{varying} architectures, which implicitly trade-off individual architectures' strengths against the ensemble's diversity and exploit architectural variation as a source of diversity. On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift. Our further analysis and ablation studies provide evidence of higher ensemble diversity due to architectural variation, resulting in ensembles that can outperform deep ensembles, even when having weaker average base learners. To foster reproducibility, our code is available: \url{https://github.com/automl/nes}
BibTeX
@inproceedings{zaidi2021neural,
  author = {Sheheryar Zaidi and Arber Zela and Thomas Elsken and Chris Holmes and Frank Hutter and Yee Whye Teh},
  title = {{Neural Ensemble Search for Uncertainty Estimation and Dataset Shift}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=HiYDAwAGWud},
}
Figure