A general framework for probabilistic model uncertainty

Vik Shirvaikar, Stephen G. Walker, Chris HolmesarXiv 2024 · DOI · arXiv

TL;DR

This paper introduces a novel framework for quantifying model uncertainty by treating "missing data" as the primary source of doubt. Instead of relying on subjective priors or complex integrations, the authors use "predictive resampling" to recursively simulate the rest of a population, allowing researchers to calculate the probability that a specific model is the "true" one based on how often it is selected across multiple simulated futures.

What Problem Does This Paper Address?

In statistics, choosing the "right" model is rarely certain. Traditionally, we handle this in two ways:

  1. Frequentist: We use a criterion (like AIC or BIC) to pick one "best" model, often ignoring the uncertainty of that choice.
  2. Bayesian: We assign "prior" probabilities to different models and calculate "posterior" probabilities using Bayes Factors.

However, Bayes Factors are notoriously sensitive to the choice of priors—small changes in your assumptions can lead to wildly different conclusions. Furthermore, standard hypothesis testing (p-values) often conflates the size of an effect with the size of the sample, leading to "statistically significant" results that have no practical meaning in large datasets. The authors seek a middle ground: a way to get the probabilistic nuance of Bayesian methods without the headache of subjective priors.

Methods

The core of the paper is an approach called Predictive Resampling. The logic is elegant: if we had access to the entire infinite population, the "true" model would be obvious. Since we only have a small sample ($x_{1:n}$), the uncertainty comes from the data we haven't seen yet ($x_{n+1:\infty}$).

The Algorithm (Algorithm 1): 1. Start with your observed data. 2. Identify the best-fitting model using a consistent selection criterion (like the Bayesian Information Criterion, or BIC). 3. Sample a new "synthetic" observation from that best model and add it to your dataset. 4. Repeat this process until you have a very large "completed" dataset. 5. Record which model is the winner for that specific "completed" population.

By running this simulation hundreds of times (Monte Carlo trials), you can see what percentage of the time each model "wins." This percentage becomes your measure of model uncertainty.

Datasets

The authors validate their framework using a mix of synthetic and classic real-world benchmarks:

  • Synthetic Normal Data: Used to demonstrate simple hypothesis testing (e.g., testing if a mean is zero).
  • Gaussian Mixture Models (GMM): Simulated data used to test "density estimation"—determining how many clusters exist in a dataset.
  • The "Galaxies" Dataset: A classic real-world dataset containing velocity measurements of 82 galaxies, used to evaluate the method's ability to identify clusters in non-simulated data.
  • High-Dimensional Linear Regression: Simulated data with 20 covariates (where only 5 are actually relevant) to test variable selection.

Key Insights and Ideas

  • Uncertainty as Missing Information: The most profound "aha moment" is the shift in perspective. Uncertainty isn't an abstract property of a parameter; it is a direct consequence of not having the full population. By "filling in" the population, we turn a static decision into a dynamic probability.
  • Consistency is Non-Negotiable: The authors highlight that you cannot use criteria like AIC or Cross-Validation for this method. These criteria are "inconsistent"—they don't necessarily pick the true model even with infinite data. You must use a consistent criterion like BIC to ensure the simulation converges.
  • Bypassing the Prior: Because the method relies on the predictive distribution of the models themselves, it eliminates the need for "subjective prior specification." This solves a major pain point in Bayesian model comparison.
  • Reframing Hypothesis Testing: It turns hypothesis testing into a decision problem. Instead of asking "How unlikely is this data under the null?", it asks "What is the probability that the null model is the best description of the entire population?"

Key Results

  • Hypothesis Testing: The method avoids the "Jeffreys-Lindley paradox," where Bayesian and Frequentist methods give contradictory results in large samples.
  • Clustering: In the "Galaxies" dataset and GMM simulations, the predictive resampling approach was much more conservative and accurate than Dirichlet Process Mixture Models (DPMM). While DPMMs tend to over-fit and suggest too many clusters as the sample size grows, the authors' method correctly converged on a simpler, more realistic number of components.
  • Variable Selection: In high-dimensional regression, the method correctly identified the five "true" variables with near-certainty as the initial sample size increased, matching the performance of complex Bayesian Gibbs samplers but with a more straightforward implementation.

Limitations and Future Work

  • Computational Cost: The method requires re-fitting models thousands of times (once for every new simulated observation across many trials). For complex, deep-learning-scale models, this would be prohibitively slow.
  • Optimization: The authors suggest that "online" model updates (updating the model incrementally as new synthetic points are added) will be necessary to scale this to higher-dimensional spaces.
  • Parallelization: While the trials are independent and easy to parallelize, the recursive nature of within a single trial remains a sequential bottleneck.

In summary, this paper provides a robust, "prior-free" way to think about model uncertainty that bridges the gap between simple decision-making and complex Bayesian inference.

Summary updated

Abstract

Existing approaches to model uncertainty typically either compare models using a quantitative model selection criterion or evaluate posterior model probabilities having set a prior. In this paper, we propose an alternative strategy which views missing observations as the source of model uncertainty, where the true model would be identified with the complete data. To quantify model uncertainty, it is then necessary to provide a probability distribution for the missing observations conditional on what has been observed. This can be set sequentially using one-step-ahead predictive densities, which recursively sample from the best model according to some consistent model selection criterion. Repeated predictive sampling of the missing data, to give a complete dataset and hence a best model each time, provides our measure of model uncertainty. This approach bypasses the need for subjective prior specification or integration over parameter spaces, addressing issues with standard methods such as the Bayes factor. Predictive resampling also suggests an alternative view of hypothesis testing as a decision problem based on a population statistic, where we directly index the probabilities of competing models. In addition to hypothesis testing, we demonstrate our approach on illustrations from density estimation and variable selection.
BibTeX
@misc{shirvaikar2024general,
  author = {Vik Shirvaikar and Stephen G. Walker and Chris Holmes},
  title = {{A general framework for probabilistic model uncertainty}},
  year = {2024},
  eprint = {2410.17108},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.17108},
}
Figure