Introducing doubt in Bayesian model comparison

Glenn D. Starkman, Roberto Trotta, Pascal M. VaudrevangearXiv 2008 · DOI · arXiv

Abstract

There are things we know, things we know we don't know, and then there are things we don't know we don't know. In this paper we address the latter two issues in a Bayesian framework, introducing the notion of doubt to quantify the degree of (dis)belief in a model given observational data in the absence of explicit alternative models. We demonstrate how a properly calibrated doubt can lead to model discovery when the true model is unknown.

Introduction

Standard Bayesian model comparison is fundamentally a relative process. Given a set of candidate models, the framework excels at identifying which model is most supported by the data. However, it suffers from a significant "blind spot": if the entire set of models is fundamentally flawed, the Bayesian evidence will still select a "winner" from the available options. It lacks an absolute measure of model adequacy.

In contrast, frequentist statistics utilizes "goodness-of-fit" tests, such as the $\chi^2$ test, to reject a model if it fails to describe the data sufficiently well, even without an explicit alternative. This paper bridges this philosophical gap by introducing statistical doubt ($D$), a formal Bayesian mechanism to represent the "unknown unknown"—the possibility that a better model exists outside our current considerations.

Methodology: Quantifying the Unknown

The core proposal is to expand the model space to include an unidentified model $\mathcal{X}$. We define Doubt ($D$) as the posterior probability of this unknown model: $D \equiv p(\mathcal{X}|d)$, where $d$ represents the observed data.

Estimating Evidence for $\mathcal{X}$

Calculating the evidence for an undefined model is non-trivial. The authors utilize the Bayesian Information Criterion (BIC) to approximate the evidence for $\mathcal{X}$. By assuming $\mathcal{X}$ is a "good" model that can achieve a high maximum likelihood, the BIC provides a proxy for its evidence based on the data's properties.

Calibration and the Doubt Ratio

To prevent "false doubt"—where statistical noise in the data leads to the erroneous rejection of a correct model—the authors introduce a calibration parameter, $\alpha$. This parameter is a quantile of the $\chi^2$ distribution that defines a "typical" realization. Furthermore, they track the Doubt Ratio ($\mathcal{R}$), defined as the ratio of posterior doubt to prior doubt. An $\mathcal{R} > 1$ indicates that the data has increased our suspicion that the current model set is incomplete.

Key Results and Discussion

The effectiveness of the doubt framework was evaluated using Gaussian linear toy models.

Correlation with Frequentist Metrics

A critical finding is the strong correlation between the Bayesian Doubt Ratio and the frequentist $\chi^2/\text{dof}$ metric. As shown in Figure 2, high values of $\chi^2/\text{dof}$ consistently correspond to high values of $\mathcal{R}$. This demonstrates that "doubt" captures the same physical intuition as traditional goodness-of-fit tests while remaining within a consistent Bayesian logical framework.

Correlation between the frequentist $\chi^2/\text{dof}$ and the Bayesian Doubt Ratio $\mathcal{R}$.

Controlling False Doubt

The calibration process ensures that the rate of false doubt can be strictly controlled. As the number of data points $n$ increases, the fraction of realizations showing false doubt converges to a pre-defined significance level $\gamma$ (e.g., 0.01 or 0.05). This behavior, illustrated in Figure 4, confirms that the method is robust against over-interpreting random fluctuations as evidence for "new physics."

Convergence of the false doubt rate to the target significance level $\gamma$ as the number of data points $n$ increases.

Model Discovery

In scenarios where the data was generated by a model omitted from the known list, the doubt $D$ surged toward 1.0 rapidly, often requiring very few data points. This suggests that high doubt can serve as a "check engine light" for scientists, signaling that the current theoretical framework is insufficient and prompting the search for new models.

Conclusion

The introduction of statistical doubt provides a rigorous way to say "none of the above" in Bayesian inference. By turning relative comparisons into absolute measures of adequacy, this framework allows for formal model discovery. Future work aims to apply this metric to complex real-world problems, such as investigating current tensions in the standard model of cosmology.

Summary updated

Abstract

There are things we know, things we know we don't know, and then there are things we don't know we don't know. In this paper we address the latter two issues in a Bayesian framework, introducing the notion of doubt to quantify the degree of (dis)belief in a model given observational data in the absence of explicit alternative models. We demonstrate how a properly calibrated doubt can lead to model discovery when the true model is unknown.
BibTeX
@misc{starkman2008introducing,
  author = {Glenn D. Starkman and Roberto Trotta and Pascal M. Vaudrevange},
  title = {{Introducing doubt in Bayesian model comparison}},
  year = {2008},
  eprint = {0811.2415},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/0811.2415},
  howpublished = {arXiv},
}
Figure