Learning from i.i.d. data under model miss-specification

Andrés R. Masegosa

Abstract

Virtually any model we use in machine learning to make predictions does not perfectly represent reality. So, most of the learning happens under model misspecification. In this work, we present a novel analysis of the generalization performance of Bayesian model averaging under model misspecification and i.i.d. data using a new family of second-order PAC-Bayes bounds. This analysis shows, in simple and intuitive terms, that Bayesian model averaging provides suboptimal generalization performance when the model is misspecified. In consequence, we provide strong theoretical arguments showing that Bayesian methods are not optimal for learning predictive models, unless the model class is perfectly specified. Using novel second-order PAC-Bayes bounds, we derive a new family of Bayesian-like algorithms, which can be implemented as variational and ensemble methods. The output of these algorithms is a new posterior distribution, different from the Bayesian posterior, which induces a posterior predictive distribution with better generalization performance. Experiments with Bayesian neural networks illustrate these findings.

The Problem: Model Misspecification

In the famous words of George Box, "all models are wrong, but some are useful." In machine learning, we almost always work under model misspecification, meaning the true data-generating process is not actually contained within our chosen family of models.

A critical flaw in standard Bayesian Model Averaging (BMA) is that it is only optimal if the model is perfectly specified. When the model is "wrong," the Bayesian posterior tends to concentrate (collapse) around the single "best" model in the family. This collapse throws away the benefits of averaging and leads to suboptimal predictions. This research aims to find a principled way to prevent this collapse and improve generalization when models are imperfect.

Methods: Second-Order PAC-Bayes Bounds

The core innovation of this work is the transition from first-order to second-order PAC-Bayes bounds. Standard Variational Inference (VI) minimizes the Evidence Lower Bound (ELBO), which is derived from a first-order Jensen inequality. This creates a linear bound that is minimized by a single point (a Dirac-delta distribution), leading to the aforementioned posterior collapse.

Theoretical comparison of Cross Entropy and the Second-order Jensen bound.

By using a second-order Jensen inequality, we derive a tighter bound that includes a variance term. This term measures the "diversity" or "disagreement" between models. Minimizing this bound naturally prevents the posterior from collapsing. This leads to two main algorithmic families:

  1. PAC$^2$-Variational Learning: Incorporates a variance term to reward model diversity.
  2. PAC$^2_T$-Variational Learning: A tighter version using a more sophisticated variance term to further bridge the gap between training loss and true cross-entropy.

This theory also extends to finite ensembles, providing a theoretical justification for why diversity is essential: it is a mathematical requirement for tighter generalization bounds under misspecification.

Key Results

The methods were evaluated on standard image datasets like Fashion-MNIST and CIFAR-10, as well as toy regression tasks. The $PAC^2$ and $PAC^2_T$ methods consistently achieved better Negative Log-Likelihood (NLL) on test data compared to standard Bayesian approaches.

Comparison of Bayesian and PAC$^2$-Variational posteriors and their predictive distributions.

As shown in the figure above, in toy linear regression examples with misspecified noise, the Bayesian posterior collapses to a single point in parameter space, failing to capture the true data distribution. In contrast, the $PAC^2$-Variational method maintains a higher-variance posterior, leading to a predictive distribution with much better coverage and lower error.

Furthermore, ensemble methods based on these bounds significantly outperformed standard ensembles. On Fashion-MNIST, the $PAC^2_T$-Ensemble showed a clear downward trend in NLL as more models were added, whereas standard methods often plateaued or improved more slowly.

Conclusion

This work demonstrates that diversity is not just a heuristic for ensembles but a fundamental requirement for optimal learning under model misspecification. By replacing first-order bounds with second-order PAC-Bayes bounds, we can derive learning objectives that naturally resist posterior collapse and yield models with superior generalization performance. Future work includes optimizing the PAC-Bayes constants and exploring more complex prior distributions to further improve performance on large-scale tasks.

Summary updated

Abstract

Virtually any model we use in machine learning to make predictions does not perfectly represent reality. So, most of the learning happens under model misspecification. In this work, we present a novel analysis of the generalization performance of Bayesian model averaging under model misspecification and i.i.d. data using a new family of second-order PAC-Bayes bounds. This analysis shows, in simple and intuitive terms, that Bayesian model averaging provides suboptimal generalization performance when the model is misspecified. In consequence, we provide strong theoretical arguments showing that Bayesian methods are not optimal for learning predictive models, unless the model class is perfectly specified. Using novel second-order PAC-Bayes bounds, we derive a new family of Bayesian-like algorithms, which can be implemented as variational and ensemble methods. The output of these algorithms is a new posterior distribution, different from the Bayesian posterior, which induces a posterior predictive distribution with better generalization performance. Experiments with Bayesian neural networks illustrate these findings.
BibTeX
@inproceedings{masegosa2020learning,
  author = {Andrés R. Masegosa},
  title = {{Learning from i.i.d. data under model miss-specification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=1XcZXRlHVe},
}
Figure