Abstract
Deep learning ensembles are widely used to improve generalization, yet the theoretical justification for their success remains a subject of debate. While Bayesian Model Averaging (BMA) is often cited as the gold standard for model combination, this paper demonstrates that BMA often fails to leverage the "cancellation of errors" effect that makes ensembles effective. By comparing uniform ensembles, Bayesian approximations, and a novel PAC-Bayesian weighting strategy based on "tandem loss," the authors show that simple uniform averages frequently outperform complex Bayesian methods. Furthermore, they demonstrate that second-order PAC-Bayesian optimization allows for the inclusion of multiple training checkpoints without the need for early stopping, providing both improved performance and nonvacuous generalization guarantees.
Introduction
Ensembling multiple neural networks is a cornerstone of modern machine learning, consistently providing better generalization and robustness than single models. The prevailing wisdom attributes this success to the "cancellation of errors": if individual models make independent mistakes, these errors average out in the aggregate.
However, a theoretical tension exists between the practice of ensembling and the theory of Bayesian inference. Bayesian Model Averaging (BMA) weights models according to their posterior probability. While mathematically rigorous for identifying the "true" underlying model, BMA is not inherently designed for model combination. This paper investigates whether the Bayesian framework is truly the optimal path for deep learning ensembles or if PAC-Bayesian theory offers a more effective alternative by explicitly accounting for model diversity.
The Limits of Bayesian Model Averaging
The authors highlight a critical property of Bayesian inference: the Bernstein-von Mises theorem. This theorem suggests that as the amount of data increases, the Bayesian posterior tends to concentrate its weight on a single "best" model. While this is ideal for model selection, it is counterproductive for ensembling. By collapsing toward a single point in the hypothesis space, BMA destroys the diversity required for error cancellation. In the limit of infinite data, a Bayesian ensemble effectively ceases to be an ensemble at all.
Methodology: Beyond Simple Averaging
The study evaluates three distinct ensembling strategies:
- Uniform Ensembles ($AVG_u$): A simple average of models trained with different random initializations. This serves as a surprisingly robust baseline.
- Bayesian Ensembles: Approximations of BMA using state-of-the-art techniques such as Cyclic Stochastic Gradient Hamiltonian Monte Carlo (cSGHMC).
- PAC-Bayesian Ensembles ($AVG_\rho$): A novel approach where ensemble weights ($\rho$) are optimized by minimizing a second-order PAC-Bayesian generalization bound.
The core innovation in the PAC-Bayesian approach is the use of tandem loss. Unlike standard loss functions that evaluate models in isolation, tandem loss measures the frequency with which two models in an ensemble fail on the same input. By minimizing a bound derived from tandem loss, the optimization process naturally favors a diverse set of models that exhibit decorrelated errors.
Experimental Setup
The authors tested these strategies across several architectures and classification tasks:
- Computer Vision: CIFAR-10 and CIFAR-100 using ResNet20, ResNet110, and WideResNet28-10.
- Medical Imaging: EyePACS (diabetic retinopathy detection) using ResNet50.
- Natural Language Processing: IMDB sentiment classification using a CNN-LSTM architecture.
Key Findings
1. The Strength of Uniform Weighting
Contrary to the assumption that Bayesian methods are superior, simple uniform ensembles of independently trained networks matched or exceeded the performance of cSGHMC. On CIFAR-100, the uniform ensemble actually outperformed the Bayesian reference by 1.3 percentage points, suggesting that the complexity of approximate Bayesian inference does not always translate to better predictive performance.
2. Leveraging Training Checkpoints
In standard training pipelines, researchers use a validation set to perform "early stopping," selecting the single best checkpoint and discarding the rest. This paper demonstrates that PAC-Bayesian weighting allows researchers to safely include all intermediate checkpoints in the ensemble. The tandem loss optimization automatically weights these checkpoints, effectively filtering out low-performing models or ensuring they do not negatively impact the collective prediction. On the IMDB dataset, this approach improved results by up to 13.3 points compared to standard methods.
3. Nonvacuous Generalization Bounds
Historically, theoretical generalization bounds for deep learning have been "vacuous," often predicting error rates higher than 100%. The second-order bounds utilized in this study are nonvacuous, providing meaningful, rigorous guarantees on test performance. For instance, on CIFAR-10, the PAC-Bayesian bound was tightened significantly through optimization, moving from 0.425 to 0.743, much closer to the observed test accuracy of ~0.93.
Discussion and Limitations
While the PAC-Bayesian approach offers significant advantages in terms of diversity and theoretical grounding, it requires a hold-out dataset to estimate tandem loss and optimize weights. In scenarios where data is extremely scarce, this requirement may be a bottleneck. Additionally, while the empirical results are strong, the theoretical bounds used are derived for majority voting; extending these specifically to weighted averages remains an area for future formalization.
In conclusion, this research challenges the dominance of BMA in deep learning. By focusing on error correlation through tandem loss and PAC-Bayesian optimization, researchers can build more robust ensembles that effectively utilize all available training information, including intermediate model checkpoints.
Summary updated