TL;DR
Neural Ensemble Search (NES) introduces a novel way to build neural network ensembles by automatically searching for and combining different architectures, rather than just retraining a single, fixed architecture with different random seeds. By leveraging architectural variation as a primary source of diversity, NES significantly improves classification accuracy, uncertainty calibration, and robustness to dataset shift compared to standard Deep Ensembles.
What Problem Does This Paper Address?
Deep learning models are notoriously overconfident when faced with "dataset shift"—situations where the data seen during deployment differs from the training data (e.g., a self-driving car encountering unexpected weather conditions).
To mitigate this, researchers typically use Deep Ensembles, which involve training the exact same neural network architecture multiple times with different random initializations. While Deep Ensembles represent the state-of-the-art for uncertainty estimation, they are constrained by their fixed architecture. They only explore the diversity that random initialization can provide within a single model's landscape. This paper asks a fundamental question: Can we achieve better diversity and robustness by automatically searching for an ensemble composed of entirely different network architectures?
Methods
The authors propose Neural Ensemble Search (NES), a framework that treats ensemble construction as a two-step optimization problem:
- Pool Building: Generate a large pool of candidate neural network architectures trained independently on the training set. The paper introduces two ways to do this:
- NES-RS (Random Search): Architectures are sampled uniformly at random from a modern Neural Architecture Search (NAS) space. It is simple and embarrassingly parallelizable.
- NES-RE (Regularized Evolution): A more guided approach based on evolutionary algorithms, where architectures are iteratively mutated over time.
- Ensemble Selection: From the pool, greedily select a subset of $M$ models using a forward step-wise selection algorithm (
ForwardSelect) without replacement to minimize validation loss.
To make the ensemble robust to dataset shift, the authors introduce a clever twist: they perform the ensemble selection step using a validation set that has been intentionally subjected to a disjoint set of data corruptions ($D_{val}^{shift}$).
Datasets
The methods were validated across two major architecture search spaces (DARTS and NAS-Bench-201) using five popular image classification datasets:
- Fashion-MNIST
- CIFAR-10
- CIFAR-100
- Tiny ImageNet
- ImageNet-16-120
To simulate dataset shift, the authors applied 19 different types of digital corruptions (like blur, noise, and weather effects) at varying severities, following the standard benchmark established by Hendrycks & Dietterich (2019).
Key Insights and Ideas
- Architectural Diversity vs. Individual Strength: Standard NAS focuses on finding the single best-performing architecture. However, NES demonstrates a fascinating trade-off: an ensemble of slightly weaker, diverse architectures can easily outperform a deep ensemble composed of the single "best" architecture.
- Function Space Exploration: Using t-SNE dimensionality reduction to visualize model predictions, the authors show that different initializations of a fixed architecture cluster closely together in function space. In contrast, varying the architecture forces the models to explore entirely different regions, yielding much higher predictive disagreement and ensemble diversity.
Key Results
- Superior Calibration and Robustness: On CIFAR-10, NES reduced the Expected Calibration Error (ECE)—a metric evaluating how well a model's confidence matches its actual accuracy—by up to 40% relative to standard Deep Ensembles under dataset shift.
- Outperforming the "Optimal" Architecture: On the NAS-Bench-201 space (where all 15,625 possible architectures are exhaustively pre-evaluated), NES ensembles consistently outperformed a Deep Ensemble made from the absolute best standalone architecture in the search space.
- Efficiency Gains: NES-RE achieved a 6.6x speedup over the random search variant (NES-RS) on NAS-Bench-201 to reach the same level of ensemble performance.
Limitations and Future Work
The primary limitation of NES is its computational cost. Building the initial pool requires training hundreds of networks from scratch. While this process is highly parallelizable, it remains resource-intensive.
The authors attempted to accelerate pool generation using popular "weight-sharing" NAS methods (where a single massive network is trained, and sub-networks are sampled from it). However, this drastically degraded performance due to "weight interference"—the sub-networks performed poorly when evaluated or retrained independently. The authors note that developing one-shot or differentiable NAS methods specifically tailored for ensemble selection is a crucial direction for future work.
Summary updated