Abstract
Bayesian model comparison (BMC) offers a principled probabilistic approach to study and rank competing models. In standard BMC, we construct a discrete probability distribution over the set of possible models, conditional on the observed data of interest. These posterior model probabilities (PMPs) are measures of uncertainty, but -- when derived from a finite number of observations -- are also uncertain themselves. In this paper, we conceptualize distinct levels of uncertainty which arise in BMC. We explore a fully probabilistic framework for quantifying meta-uncertainty, resulting in an applied method to enhance any BMC workflow. Drawing on both Bayesian and frequentist techniques, we represent the uncertainty over the uncertain PMPs via meta-models which combine simulated and observed data into a predictive distribution for PMPs on new data. We demonstrate the utility of the proposed method in the context of conjugate Bayesian regression, likelihood-based inference with Markov chain Monte Carlo, and simulation-based inference with neural networks.
Introduction
In Bayesian statistics, researchers compare competing models using Posterior Model Probabilities (PMPs). If a model is assigned a PMP of 0.9, it is standard practice to conclude there is a 90% probability that the model is the "correct" one within the candidate set. However, PMPs are notoriously sensitive to small changes in data, especially when sample sizes are limited—a phenomenon often described as the "dance of Bayes factors."
This sensitivity leads to overconfidence: standard methods may signal a near-certain preference for one model even when the data is sparse or the model is misspecified. Currently, researchers lack a standard metric to ask, "How much should I trust this specific probability?" This paper addresses this gap by introducing and quantifying meta-uncertainty—the uncertainty inherent in the PMPs themselves.
Conceptualizing Levels of Uncertainty
The authors propose a hierarchical framework to describe three distinct levels of uncertainty in the model comparison process:
- Level 1 (Standard PMP): The uncertainty regarding which model is true, given a specific observed dataset.
- Level 2 (Model-Implied Variation): The variation in PMPs that would be expected if new datasets were collected from the same underlying process.
- Level 3 (Meta-Model Uncertainty): The uncertainty in estimating the Level 2 distribution due to the use of a finite number of simulations.
The Meta-Uncertainty Framework
To quantify these levels, the authors developed a simulation-based workflow. For every candidate model in the set, thousands of synthetic datasets are generated. PMPs are then calculated for each synthetic dataset, creating a "reference distribution" of what PMPs should look like if a specific model were indeed the data-generating process.
A Bayesian "meta-model" (such as a Dirichlet or Logistic Normal distribution) is then fitted to these simulated PMPs. Finally, the observed PMP from the real data is combined with these meta-models to create a predictive mixture distribution. This distribution represents the PMPs a researcher should expect to see in a future replication study, providing a robust measure of the stability of the scientific conclusion.
Mitigating Overconfidence
A key insight of this framework is its ability to act as a "simulation-based correction." If real-world data yields a PMP of 0.99, but simulations show that the true model typically only yields a PMP of 0.60, the framework flags the result as "atypical" and potentially overconfident.
In toy conjugate model experiments, the authors demonstrated that when data contains outliers (atypical data), standard PMPs can erroneously shoot to ~1.0 for a misspecified model. The meta-uncertainty framework correctly adjusted this back to approximately 0.55, reflecting the true underlying evidence.
Applications and Results
The utility of the method was validated across three diverse scenarios:
- Conjugate Regression: Demonstrating how meta-uncertainty behaves and shrinks as sample size ($N$) increases.
- Epidemiology: Analyzing a 1978 H1N1 outbreak. The framework confirmed that a high PMP favoring the SIR model was consistent with model expectations, increasing confidence in the result.
- Neural Simulators: Using BayesFlow (amortized deep learning) to calculate PMPs for complex Hodgkin-Huxley equations. This proved that meta-uncertainty can be efficiently calculated for high-dimensional models without closed-form likelihoods.
Conclusion
By shifting the focus from a single point estimate to the stability and replicability of model probabilities, the meta-uncertainty framework provides a vital reality check for Bayesian inference. While computational costs for simulations remain a consideration, the use of amortized inference methods offers a path forward for complex, real-world applications. Future work will explore extending this framework to "open-world" scenarios where the true model may not be present in the candidate set.
Summary updated