Learning under Model Misspecification: Applications to Variational and Ensemble methods

Andrés R. Masegosa

TL;DR: Standard Bayesian methods often perform poorly because they assume your model family contains the "true" reality. This paper introduces a new framework using second-order PAC-Bayes bounds that explicitly accounts for model imperfection, leading to new variational and ensemble algorithms that prioritize model diversity and achieve superior generalization.

The Problem with Being "Right"

In machine learning, we often live by George Box’s famous aphorism: "All models are wrong, but some are useful." However, standard Bayesian statistics is built on the assumption that our model family is "well-specified"—meaning the true data-generating process is actually hidden somewhere inside our hypothesis space.

When this assumption fails (which is almost always the case in practice), Bayesian Model Averaging (BMA) behaves strangely. Instead of providing a robust average over many plausible models, the posterior distribution tends to collapse around the single "least-wrong" model. This "spike" ignores the uncertainty created by the model's inherent flaws, leading to suboptimal predictions and overconfidence. The fundamental question then becomes: How should we learn when we know our model is fundamentally incapable of representing reality perfectly?

Bridging PAC-Bayes and Bayesian Inference

To address this, the authors bridge the gap between PAC-Bayesian theory—which provides frequentist guarantees on generalization—and Bayesian inference.

Standard Variational Inference (VI) can be viewed as minimizing a "first-order" PAC-Bayes bound. Because this bound is linear with respect to the model predictions, its mathematical "urge" is to push the solution to the corners, resulting in a concentrated, non-diverse posterior.

Theoretical bounds under model misspecification, illustrating the gap between expected log-loss and cross-entropy, and the proposed second-order Jensen bound.

To counter this, the paper derives a new family of second-order bounds. These bounds include a crucial variance term ($\mathbb{V}$) that measures the "disagreement" or diversity among the models in a distribution. This leads to the $PAC^2$ framework. The proposed $PAC^2$ objective resembles the standard Evidence Lower Bound (ELBO) used in VI, but with an added penalty term that accounts for the variance of the predictions. This term acts as a "safety buffer" against misspecification.

The $PAC^2$ Framework: Variational and Ensemble Learning

Using these second-order bounds, the authors develop two new algorithmic approaches:

  1. $PAC^2$-Variational Learning: A modified version of VI where the objective function rewards distributions that fit the data well and maintain a healthy level of internal disagreement.
  2. $PAC^2$-Ensemble Learning: An adaptation for finite sets of models. This provides a formal theoretical justification for why "diversity" is the secret sauce in successful ensemble methods like Random Forests or Deep Ensembles. Diversity is not just a heuristic; it is a mathematical requirement for minimizing the gap between training error and true generalization error when models are wrong.
Comparison of standard Bayesian and PAC^2-Variational methods. The Bayesian posterior collapses to a spike, leading to narrow predictive uncertainty. The PAC^2 posterior maintains diversity, yielding better calibrated epistemic uncertainty.

As seen in the comparison above, preventing the posterior from collapsing into a "spike" forces it to remain a broader distribution that better covers the true data distribution, ultimately yielding more reliable predictive uncertainty.

Experimental Validation

The methods were evaluated on standard computer vision benchmarks, including Fashion-MNIST and CIFAR-10, for both standard classification and self-supervised "image completion" tasks. For the self-supervised tasks, the authors intentionally used misspecified data models (Normal and Binomial distributions) to test how the algorithms handled predictive noise.

Key Results: * Better Generalization: In almost all tests, the $PAC^2$ methods achieved lower Negative Log-Likelihood (NLL) than standard Bayesian methods. On Fashion-MNIST, the $PAC^2$-Variational approach showed a ~5% improvement in NLL over a Maximum A Posteriori (MAP) model. * Ensemble Superiority: The ensemble versions of the $PAC^2$ algorithms significantly outperformed the variational versions. Mixtures of "spikes" (Dirac-deltas) are more flexible and can represent complex, multimodal distributions that a simple Gaussian variational posterior cannot. * Robustness to Noise: The $PAC^2$ methods were particularly effective in the image completion tasks, where the model's assumptions about noise were intentionally misspecified.

Limitations and Future Directions

While promising, the $PAC^2$ framework has limitations. On the CIFAR-10 classification task, the variational approaches did not outperform the MAP model, likely due to poor prior specification. If the starting assumptions are too far off, even a better bound struggles to compensate.

Furthermore, the experiments primarily used "mean-field" (fully factorized) distributions. More complex variational families, such as normalizing flows, might be needed to fully unlock the potential of $PAC^2$ learning. Finally, calculating the variance term across all pairs of models in an ensemble adds computational complexity compared to standard Stochastic Gradient Descent, though the paper does present scalable approximations.

Summary updated

Abstract

Virtually any model we use in machine learning to make predictions does not perfectly represent reality. So, most of the learning happens under model misspecification. In this work, we present a novel analysis of the generalization performance of Bayesian model averaging under model misspecification and i.i.d. data using a new family of second-order PAC-Bayes bounds. This analysis shows, in simple and intuitive terms, that Bayesian model averaging provides suboptimal generalization performance when the model is misspecified. In consequence, we provide strong theoretical arguments showing that Bayesian methods are not optimal for learning predictive models, unless the model class is perfectly specified. Using novel second-order PAC-Bayes bounds, we derive a new family of Bayesian-like algorithms, which can be implemented as variational and ensemble methods. The output of these algorithms is a new posterior distribution, different from the Bayesian posterior, which induces a posterior predictive distribution with better generalization performance. Experiments with Bayesian neural networks illustrate these findings.
BibTeX
@inproceedings{masegosa2020learning,
  author = {Andrés R. Masegosa},
  title = {{Learning under Model Misspecification: Applications to Variational and Ensemble methods}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=1XcZXRlHVe},
}
Figure