Bayesian vs. PAC-Bayesian Deep Neural Network Ensembles

Nick Hauptvogel, Christian Igel

Abstract

Deep learning ensembles are widely used to improve generalization, yet the theoretical justification for their success remains a subject of debate. While Bayesian Model Averaging (BMA) is often cited as the gold standard for model combination, this paper demonstrates that BMA often fails to leverage the "cancellation of errors" effect that makes ensembles effective. By comparing uniform ensembles, Bayesian approximations, and a novel PAC-Bayesian weighting strategy based on "tandem loss," the authors show that simple uniform averages frequently outperform complex Bayesian methods. Furthermore, they demonstrate that second-order PAC-Bayesian optimization allows for the inclusion of multiple training checkpoints without the need for early stopping, providing both improved performance and nonvacuous generalization guarantees.

Introduction

Ensembling multiple neural networks is a cornerstone of modern machine learning, consistently providing better generalization and robustness than single models. The prevailing wisdom attributes this success to the "cancellation of errors": if individual models make independent mistakes, these errors average out in the aggregate.

However, a theoretical tension exists between the practice of ensembling and the theory of Bayesian inference. Bayesian Model Averaging (BMA) weights models according to their posterior probability. While mathematically rigorous for identifying the "true" underlying model, BMA is not inherently designed for model combination. This paper investigates whether the Bayesian framework is truly the optimal path for deep learning ensembles or if PAC-Bayesian theory offers a more effective alternative by explicitly accounting for model diversity.

The Limits of Bayesian Model Averaging

The authors highlight a critical property of Bayesian inference: the Bernstein-von Mises theorem. This theorem suggests that as the amount of data increases, the Bayesian posterior tends to concentrate its weight on a single "best" model. While this is ideal for model selection, it is counterproductive for ensembling. By collapsing toward a single point in the hypothesis space, BMA destroys the diversity required for error cancellation. In the limit of infinite data, a Bayesian ensemble effectively ceases to be an ensemble at all.

Methodology: Beyond Simple Averaging

The study evaluates three distinct ensembling strategies:

  1. Uniform Ensembles ($AVG_u$): A simple average of models trained with different random initializations. This serves as a surprisingly robust baseline.
  2. Bayesian Ensembles: Approximations of BMA using state-of-the-art techniques such as Cyclic Stochastic Gradient Hamiltonian Monte Carlo (cSGHMC).
  3. PAC-Bayesian Ensembles ($AVG_\rho$): A novel approach where ensemble weights ($\rho$) are optimized by minimizing a second-order PAC-Bayesian generalization bound.

The core innovation in the PAC-Bayesian approach is the use of tandem loss. Unlike standard loss functions that evaluate models in isolation, tandem loss measures the frequency with which two models in an ensemble fail on the same input. By minimizing a bound derived from tandem loss, the optimization process naturally favors a diverse set of models that exhibit decorrelated errors.

Experimental Setup

The authors tested these strategies across several architectures and classification tasks:

  • Computer Vision: CIFAR-10 and CIFAR-100 using ResNet20, ResNet110, and WideResNet28-10.
  • Medical Imaging: EyePACS (diabetic retinopathy detection) using ResNet50.
  • Natural Language Processing: IMDB sentiment classification using a CNN-LSTM architecture.

Key Findings

1. The Strength of Uniform Weighting

Contrary to the assumption that Bayesian methods are superior, simple uniform ensembles of independently trained networks matched or exceeded the performance of cSGHMC. On CIFAR-100, the uniform ensemble actually outperformed the Bayesian reference by 1.3 percentage points, suggesting that the complexity of approximate Bayesian inference does not always translate to better predictive performance.

2. Leveraging Training Checkpoints

In standard training pipelines, researchers use a validation set to perform "early stopping," selecting the single best checkpoint and discarding the rest. This paper demonstrates that PAC-Bayesian weighting allows researchers to safely include all intermediate checkpoints in the ensemble. The tandem loss optimization automatically weights these checkpoints, effectively filtering out low-performing models or ensuring they do not negatively impact the collective prediction. On the IMDB dataset, this approach improved results by up to 13.3 points compared to standard methods.

3. Nonvacuous Generalization Bounds

Historically, theoretical generalization bounds for deep learning have been "vacuous," often predicting error rates higher than 100%. The second-order bounds utilized in this study are nonvacuous, providing meaningful, rigorous guarantees on test performance. For instance, on CIFAR-10, the PAC-Bayesian bound was tightened significantly through optimization, moving from 0.425 to 0.743, much closer to the observed test accuracy of ~0.93.

Discussion and Limitations

While the PAC-Bayesian approach offers significant advantages in terms of diversity and theoretical grounding, it requires a hold-out dataset to estimate tandem loss and optimize weights. In scenarios where data is extremely scarce, this requirement may be a bottleneck. Additionally, while the empirical results are strong, the theoretical bounds used are derived for majority voting; extending these specifically to weighted averages remains an area for future formalization.

In conclusion, this research challenges the dominance of BMA in deep learning. By focusing on error correlation through tandem loss and PAC-Bayesian optimization, researchers can build more robust ensembles that effectively utilize all available training information, including intermediate model checkpoints.

Summary updated

Abstract

It is common practice to combine deep neural networks to ensembles. These deep ensembles can profit from the cancellation of errors effect: Errors by ensemble members may average out and the ensemble achieves better generalization performance than each individual network. Bayesian neural networks learn a posterior distribution over model parameters, and sampling and weighting networks according to this posterior yields an ensemble model referred to as Bayes ensemble. In this study, we stress that neither the sampling nor the weighting in a Bayes ensemble are particularly well-suited for increasing generalization performance, as they do not support the cancellation of errors effect, which is evident in the limit from the Bernstein-von~Mises theorem for misspecified models. In contrast, a weighted average of models, where the weights are optimized by minimizing a PAC-Bayesian generalization bound, can improve generalization performance. This requires that the optimization takes correlations between models into account, which can be achieved by minimizing the tandem loss at the cost that hold-out data for estimating error correlations need to be available. The PAC-Bayesian weighting increases the robustness against correlated models and models with lower performance in an ensemble. This allows us to safely add several models from the same learning process to an ensemble, instead of using early-stopping for selecting a single weight configuration. Our study presents empirical results supporting these conceptual considerations on four different classification datasets. We show that state-of-the-art Bayes ensembles from the literature, despite being computationally demanding, do not improve over simple uniformly weighted deep ensembles and cannot match the performance of deep ensembles weighted by optimizing the tandem loss, which additionally come with non-vacuous generalization guarantees.
BibTeX
@misc{hauptvogel2024bayesian,
  author = {Nick Hauptvogel and Christian Igel},
  title = {{Bayesian vs. PAC-Bayesian Deep Neural Network Ensembles}},
  year = {2024},
  eprint = {2406.05469},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.05469},
}
Figure