Quantifying Statistical Evidence with Bayesian Model Comparison

161 concepts136 papers2 proposals

In scientific research, the dominant paradigm for verifying hypotheses has long been Null Hypothesis Significance Testing (NHST). However, as argued by Daniel Khashabi et al. in Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses, frequentist $p$-values are widely misunderstood and force researchers into a binary "significant vs. non-significant" mindset.

Crucially, a high $p$-value does not mean the null hypothesis is true; it merely means the data failed to reject it. This leaves researchers unable to distinguish between an underpowered study with inconclusive data and a genuine absence of an effect. Bayesian model comparison resolves this fundamental limitation by shifting the question from "How unlikely is this data under a single null hypothesis?" to "Which of our competing models best predicted the observed data?"

The Core Mechanism: Marginal Likelihood

To compare competing hypotheses, the Bayesian framework evaluates the marginal likelihood (also known as the model evidence). For a given model $M$ governed by parameters $\theta$, the marginal likelihood is computed by integrating the likelihood of the data over the entire prior distribution of the parameters:

$$P(D|M) = \int P(D|\theta, M)P(\theta|M)d\theta$$

Instead of evaluating a model solely at its single best-fitting parameter combination (the maximum likelihood), the marginal likelihood considers the average performance of the model across all parameter values allowed by its priors. It answers a simple question: How well did the model predict the data before it was observed?

When comparing two competing hypotheses—such as a null hypothesis $M_0$ and an alternative hypothesis $M_1$—we compute the ratio of their marginal likelihoods. This ratio is known as the Bayes Factor ($BF_{12}$):

$$BF_{12} = \frac{P(D|M_1)}{P(D|M_2)}$$

As reviewed by Zhengxiao Wei et al. in A Review of Bayesian Hypothesis Testing and Its Practical Implementations, the Bayes Factor provides a continuous scale of relative evidence. A Bayes factor of 1 means both models predicted the data equally well, while a Bayes factor of 10 indicates the data is ten times more probable under $M_1$ than $M_2$.

Why It Works: The Geometry of Occam's Razor

The reason integrating over the parameter space is so powerful is that it introduces an automatic mathematical penalty for complexity, often referred to as volume effects.

Imagine a simple model with a narrow prior space and a complex model with many parameters and a broad prior space. Because the complex model is highly flexible, it can adapt to explain a vast array of potential datasets. However, because probabilities must integrate to 1, its prior probability mass must be spread thinly across a massive volume of parameter space. Consequently, its predictive density for any specific dataset is diluted.

Conversely, a simple model concentrates its prior mass on a small, specific set of outcomes. If the observed data falls within that narrow window, the simple model wins because its average predictive density is much higher than that of the diluted, complex model.

Predictive Density P(D|M)
       ^
       |       / \  <-- Simple Model (Concentrated prior mass)
       |      /   \
       |    _/.____.\_  <-- Complex Model (Diluted prior mass)
       +-----------------------------------> Possible Datasets

The authors of Unveiling ν secrets with cosmological data: Neutrino masses and mass hierarchy leverage this property to calculate posterior odds between discrete physical hierarchies, demonstrating how Bayesian model comparison naturally accounts for the available parameter space.

In machine learning, Alexander Immer et al. show in Probing as Quantifying Inductive Bias that evaluating neural representations via marginal likelihood automatically accounts for probe complexity. This ensures that a representation is judged on its intrinsic quality rather than the capacity of an overly flexible probe to overfit the task.

Actively Confirming the Null

Because the Bayes Factor is a symmetric ratio, it can quantify evidence for the null hypothesis just as easily as it can for an alternative hypothesis.

In psychological science, John Lurquin et al. used this framework in No Evidence of the Ego-Depletion Effect across Task Characteristics and Individual Differences: A Pre-Registered Study to demonstrate that data actively supported the absence of the "ego-depletion" effect, rather than leaving the result in statistical limbo.

Similarly, Anuj Shukla demonstrated in Numerical Magnitude Affects Accuracy but Not Precision of Temporal Judgments that numerical magnitude has zero effect on the precision of internal clocks, providing a definitive measure of a null effect where frequentist methods could only fail to reject the null.

Comparing Non-Nested Hypotheses

In frequentist statistics, comparing models that are not "nested"—meaning one model cannot be derived as a special case of the other—is highly problematic. Bayesian model comparison handles this effortlessly because every hypothesis is treated as a generative model that defines a prior distribution over the data.

For example, Philipp Singer et al. developed HypTrails: A Bayesian Approach for Comparing Hypotheses About Human Trails on the Web to compare completely different behavioral theories of web navigation. By encoding distinct hypotheses (such as geographic proximity versus semantic similarity) as informative Dirichlet priors, the authors could directly rank which underlying transition structure best explained actual human activity sequences.

Overcoming Computational Intractability

The primary challenge of Bayesian model comparison is that calculating the marginal likelihood integral is often analytically impossible for complex, non-linear models. To overcome this, researchers rely on advanced numerical sampling techniques.

In SYSBIONS: nested sampling for systems biology, Michael Stumpf demonstrates how nested sampling can be used to efficiently compute model evidence for complex biological networks. By transforming the multi-dimensional evidence integral into a one-dimensional integral over the prior element, nested sampling bypasses the inefficiencies of standard Monte Carlo methods. This makes rigorous model comparison possible even in highly non-linear, high-dimensional spaces, allowing researchers to confidently select the most appropriate model from a set of competing biological hypotheses.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bayesian_model_comparison_92c19f66,
  title = {Bayesian model comparison},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bayesian-model-comparison/rdr_92c19f668d28484b8f463326c8cc9fc1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bayesian_model_comparison_92c19f66_v1,
  title = {Bayesian model comparison},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bayesian-model-comparison/rdr_92c19f668d28484b8f463326c8cc9fc1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake