TL;DR
This paper introduces Neural Ensemble Search (NES), a novel method for automatically constructing ensembles of neural networks with diverse architectures. NES significantly outperforms traditional deep ensembles in accuracy, uncertainty estimation, and robustness to dataset shift, highlighting the importance of architectural diversity.
What Problem Does This Paper Address?
Deep neural networks are increasingly used in critical applications like medical imaging and autonomous driving, where not only accurate predictions but also reliable uncertainty estimates and robustness to dataset shift are crucial. While deep ensembles (training multiple instances of the same network architecture with different random initializations) have shown state-of-the-art performance in these areas, they primarily rely on weight diversity. This paper addresses the limitation of deep ensembles by proposing a method to actively search for and combine diverse architectures within an ensemble, aiming to achieve even better performance, calibration, and robustness.
Methods
The core contribution is Neural Ensemble Search (NES), a framework that optimizes the architectures of base learners within an ensemble to minimize ensemble loss. NES operates in two main steps:
- Pool Building: A pool of potential base learners is created. The paper proposes two strategies for this:
- NES with Random Search (NES-RS): Architectures are sampled uniformly from a search space and trained.
- NES with Regularized Evolution (NES-RE): An evolutionary algorithm inspired by regularized evolution is used to explore the architecture search space, generating a diverse pool of trained networks.
- Ensemble Selection: A subset of base learners is selected from the pool to form the final ensemble. The paper uses ForwardSelect, a greedy, step-wise selection method, which iteratively adds the network that most improves the ensemble's performance on a validation set. Crucially, for dataset shift robustness, a shifted validation set can be used during selection.
The key novelty lies in actively searching for diverse architectures, rather than relying solely on random initializations of a fixed architecture.
Datasets
The paper evaluates its methods on several standard image classification datasets:
- Fashion-MNIST
- CIFAR-10
- CIFAR-100
- Tiny ImageNet
- ImageNet-16-120
It also utilizes dataset shifts from Hendrycks & Dietterich [27] for evaluating robustness. The architecture search spaces used are DARTS [40] and NAS-Bench-201 [13].
Key Insights and Ideas
- Architectural Diversity is Key: The paper strongly suggests that diversity in network architectures, not just weights, is crucial for building high-performing, robust, and well-calibrated ensembles.
- NES Outperforms Deep Ensembles: By actively searching for diverse architectures, NES consistently achieves better results than deep ensembles, even when the average base learner in the deep ensemble is stronger.
- Dataset Shift Robustness: NES, particularly when adapted to use shifted validation data during ensemble selection, demonstrates superior robustness to dataset shift compared to deep ensembles.
- Ensemble Selection Matters: The choice of ensemble selection strategy (like ForwardSelect) plays a significant role in leveraging the diversity of the base learners.
Key Results
- Improved Performance: NES ensembles consistently outperform deep ensembles in terms of Negative Log-Likelihood (NLL), classification error, and Expected Calibration Error (ECE) across various datasets and search spaces. For instance, on CIFAR-10 with dataset shift, NES achieved up to a 7 percentage point reduction in error.
- Enhanced Uncertainty Calibration: NES ensembles show better uncertainty calibration, especially under dataset shift.
- Robustness to Dataset Shift: NES demonstrates significant improvements in robustness to dataset shift, outperforming baselines by a notable margin.
- Diversity Metrics: Ensembles constructed by NES exhibit higher diversity, as measured by predictive disagreement and oracle ensemble loss, compared to deep ensembles.
- Computational Efficiency: While NES involves searching for architectures, NES-RS is easily parallelizable, similar to deep ensembles. NES-RE, though more guided, also offers competitive performance.
Limitations and Future Work
- Computational Cost: The primary limitation acknowledged is the computational cost associated with building the pool of base learners, especially for large search spaces.
- Future Work: The authors suggest exploring differentiable NAS methods for more efficient pool generation and investigating other hyperparameters (beyond architecture) for ensemble optimization. They also highlight the potential for applying NES to improve existing ensemble systems and its implications for safe AI deployment.
Summary updated