A framework for probabilistic inferences from imperfect models

Meng Li, David B. DunsonarXiv 2016 · arXiv

Introduction

In statistical modeling, the Bayesian paradigm offers a structured approach to uncertainty through posterior model probabilities. However, this framework traditionally operates under the "$\mathcal{M}$-closed" assumption—the belief that the true data-generating process is contained within the set of candidate models. In practice, as George Box famously noted, "all models are wrong." When we operate in an "$\mathcal{M}$-open" environment where every candidate model is imperfect, traditional Bayesian Model Averaging (BMA) often fails. It tends to become overconfident, assigning nearly all weight to the "least bad" model, and exhibits extreme sensitivity to the choice of prior distributions.

This paper introduces a new concept: absolute model probabilities, or D-probabilities. By measuring the quality of models relative to a flexible nonparametric reference rather than just against each other, this framework provides a more robust measure of model adequacy and a natural penalty for complexity.

The D-Probability Framework

The core innovation of this research is the shift from relative model comparison to absolute assessment. Instead of assuming one candidate model is true, the researchers compare each parametric model to a "gold standard" or "oracle"—a surrogate for the true data-generating process.

1. The Nonparametric Reference

Since the true process is unknown, the framework employs a Nonparametric Bayesian model as a reference. In the context of linear regression, a Gaussian Process (GP) is used. Because GPs are highly flexible and can capture complex, non-linear patterns, they serve as an effective proxy for the underlying truth that simple parametric models might miss.

2. Measuring Divergence

The "distance" between a candidate model and the nonparametric reference is quantified using Kullback-Leibler (KL) divergence. The D-probability for a specific model is then defined by exponentiating the negative KL divergence: $$\pi_j = \exp\{-n \text{KL}(\text{Reference}, \text{Model}_j)\}$$

This formulation has a striking mathematical parallel to the Boltzmann-Gibbs distribution in statistical mechanics, where the KL divergence acts as the "energy" of the system.

3. Automatic Complexity Penalty

Unlike traditional metrics like AIC or BIC, which add an explicit penalty term based on the number of parameters, D-probabilities incorporate a complexity penalty naturally. The mathematics of the KL divergence inherently accounts for the "spread" of the model, penalizing overly complex models that do not provide a commensurate increase in fit quality.

Key Advantages

Absolute vs. Relative Adequacy

Standard BMA can only tell you if Model A is better than Model B. If both are terrible, BMA will still pick a "winner." D-probabilities, however, provide an absolute measure. If the D-probabilities for all models in a set are near zero, it serves as a quantitative signal that the entire model class is inadequate for the data at hand.

Robustness to Priors

One of the most significant practical benefits is prior robustness. Traditional BMA is notoriously sensitive to the choice of parameter priors, often requiring careful tuning to avoid arbitrary results. D-probabilities allow for the use of "default" or even improper priors without the typical mathematical breakdowns, making the method more objective and easier for practitioners to implement.

Empirical Applications

Simulation Studies

The researchers tested the framework across various scenarios, including data generated from linear, logarithmic, and oscillating (sine wave) functions. The D-probabilities correctly identified when the "true" model was missing from the candidate list and avoided the overconfidence typical of BMA in those scenarios.

The Ozone Dataset

The framework was applied to a classic dataset of 330 daily ozone readings in Los Angeles. When evaluating various linear models against a Gaussian Process reference, the absolute D-probabilities for all linear models were extremely small (approximately $10^{-22}$). This provided clear evidence that linear models were fundamentally insufficient for this data. Furthermore, in out-of-sample prediction tasks, D-probabilities outperformed common alternatives like Exponential Weighting by distributing weights across a more diverse set of "good" models, effectively "hedging bets" rather than collapsing to a single, potentially flawed model.

Conclusion and Future Work

D-probabilities offer a rigorous way to acknowledge model imperfection while still making useful probabilistic inferences. By moving away from the $\mathcal{M}$-closed assumption, this framework provides a more honest assessment of model fit and a more stable foundation for model averaging.

Future research directions include:

  • Computational Efficiency: Reducing the overhead of fitting nonparametric reference models for very large datasets.
  • Discrete Data: Extending the framework to categorical data using Dirichlet Process Mixtures.
  • High-Dimensionality: Investigating how the "oracle" surrogate approach scales when the number of predictors significantly exceeds the number of observations.

Summary updated

Abstract

The Bayesian paradigm provides a natural way to deal with uncertainty in model selection through assigning each model in a list of models under consideration a posterior probability. Unfortunately, this framework relies on the assumption that one of the models in the list is the true model. When this assumption is violated and all the models are imperfect, interpretation of posterior model probabilities is unclear. We propose a new concept of {\em absolute} model probabilities, which measure the quality of imperfect models. This concept leads to divergence-based estimates ({\em D-probabilities}) relying on evaluating parametric models relative to a nonparametric Bayesian reference using Kullback-Leibler divergence. While providing good-of-fit assessment, D-probabilities avoid some of the pitfalls of usual posterior model probabilities including large sensitivity to prior choice. In an application to linear model selection against a Gaussian process reference, we provide simple analytic forms for routine implementation and show that D-probabilities automatically penalize model complexity. Some asymptotic properties of this framework are described. Absolute model probabilities have several interesting probabilistic interpretations, and can potentially be applied in broad problems. The framework is illustrated through simulations and applications.
BibTeX
@misc{li2016framework,
  author = {Meng Li and David B. Dunson},
  title = {{A framework for probabilistic inferences from imperfect models}},
  year = {2016},
  eprint = {1611.01241},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.01241},
}
Figure