Meta-Uncertainty in Bayesian Model Comparison

Marvin Schmitt, Stefan T. Radev, Paul-Christian Bürkner

Abstract: Bayesian model comparison (BMC) offers a principled probabilistic approach to study and rank competing models. In standard BMC, we construct a discrete probability distribution over the set of possible models, conditional on the observed data of interest. These posterior model probabilities (PMPs) are measures of uncertainty, but—when derived from a finite number of observations—are also uncertain themselves. In this paper, we conceptualize distinct levels of uncertainty which arise in BMC. We explore a fully probabilistic framework for quantifying meta-uncertainty, resulting in an applied method to enhance any BMC workflow. Drawing on both Bayesian and frequentist techniques, we represent the uncertainty over the uncertain PMPs via meta-models which combine simulated and observed data into a predictive distribution for PMPs on new data. We demonstrate the utility of the proposed method in the context of conjugate Bayesian regression, likelihood-based inference with Markov chain Monte Carlo, and simulation-based inference with neural networks.

Introduction

Bayesian Model Comparison (BMC) is a cornerstone of probabilistic inference, allowing researchers to weigh evidence between competing hypotheses. The primary output of BMC is the Posterior Model Probability (PMP), which quantifies the likelihood of a model being correct given the observed data. However, PMPs are often treated as static truths rather than random variables. In practice, a small change in the dataset can lead to drastically different PMPs—a phenomenon known as the "dance of Bayes factors." This inherent variability suggests a need for meta-uncertainty: uncertainty about the uncertainty itself.

Standard BMC often suffers from overconfidence, where researchers may select a model based on a high PMP that is actually an artifact of a specific, "atypical" dataset. This paper introduces a hierarchical framework that uses simulations to quantify how much these probabilities are expected to vary, providing a principled way to gauge the replicability of model selection results.

A Hierarchical Framework for Meta-Uncertainty

To address the overconfidence and replicability issues in standard BMC, the authors propose a three-level hierarchy of uncertainty. This framework contextualizes observed PMPs within the broader distribution of PMPs that could have been observed under the candidate models.

The meta-uncertainty framework: from data generation to predictive mixture.

The framework is structured as follows:

  • Level 1 (Standard PMP): Represents the standard uncertainty within a single PMP vector (how much models differ for one dataset).
  • Level 2 (Aleatoric Variation): Captures the variation of PMPs across many different datasets generated by the same model.
  • Level 3 (Epistemic Uncertainty): Accounts for the uncertainty regarding the parameters of these PMP distributions, arising because we can only run a finite number of simulations.

The workflow involves simulating thousands of "model-implied" datasets for each candidate model and calculating their PMPs. These simulated PMPs are then used to fit a "meta-model" (such as a Dirichlet or Logistic Normal distribution). Finally, the observed PMP from real data is combined with these meta-models to create a predictive mixture distribution, representing the range of PMPs a researcher should expect to see in a future replication study.

Mitigating Overconfidence in Atypical Data

A significant advantage of this framework is its ability to correct for "atypical" datasets. In standard BMC, an outlier dataset might yield a high PMP for a specific model, leading to overconfident conclusions. The meta-uncertainty framework acts as a "simulation-based correction," pulling the result back toward a more modest, realistic expectation based on the simulation distributions.

Comparison of PMP distributions for typical and atypical data.

As shown in the figure above, when data is atypical (e.g., an extreme outlier), the predictive mixture mean is significantly lower than the raw PMP, signaling that the result may be an artifact. Conversely, for typical data, the predictive mixture aligns closely with the observed PMP, reinforcing the robustness of the conclusion. This "pull" ensures that the method doesn't introduce permanent bias; as the number of observations ($N$) increases, meta-uncertainty vanishes, and the method converges to standard results.

Applications and Scalability

The framework was evaluated across three distinct scenarios:

  1. Toy Conjugate Model: A synthetic Bayesian linear regression setup used to provide a "ground truth" for how uncertainty levels behave as sample size increases.
  2. Epidemiological Modeling: An analysis of a 1978 H1N1 influenza outbreak using MCMC. The framework confirmed that the high PMP for the SIR model was robust and aligned with simulation predictions.
  3. Neural Activity (Simulation-Based Inference): Using BayesFlow (a deep learning approach), the authors showed that amortized neural estimators allow this "meta" analysis to be performed in a fraction of a second, even for complex models with intractable likelihoods.

Conclusion

Meta-uncertainty provides a principled way to gauge the replicability of Bayesian model selection. By moving beyond a single PMP vector to a predictive distribution, researchers can better communicate the reliability of their findings. While the current framework assumes a "closed world" (the true model is among the candidates), it offers a scalable, model-agnostic tool to enhance any BMC workflow.

Summary updated

Abstract

Bayesian model comparison (BMC) offers a principled probabilistic approach to study and rank competing models. In standard BMC, we construct a discrete probability distribution over the set of possible models, conditional on the observed data of interest. These posterior model probabilities (PMPs) are measures of uncertainty, but—when derived from a finite number of observations—are also uncertain themselves. In this paper, we conceptualize distinct levels of uncertainty which arise in BMC. We explore a fully probabilistic framework for quantifying meta-uncertainty, resulting in an applied method to enhance any BMC workflow. Drawing on both Bayesian and frequentist techniques, we represent the uncertainty over the uncertain PMPs via meta-models which combine simulated and observed data into a predictive distribution for PMPs on new data. We demonstrate the utility of the proposed method in the context of conjugate Bayesian regression, likelihood-based inference with Markov chain Monte Carlo, and simulation-based inference with neural networks.
BibTeX
@inproceedings{schmitt2023meta,
  author = {Marvin Schmitt and Stefan T. Radev and Paul-Christian Bürkner},
  title = {{Meta-Uncertainty in Bayesian Model Comparison}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=lMJqVcYZXN},
}
Figure