Quantifying Evidence for the Null Hypothesis with Bayes Factors

This direction is merged into another direction: Bayesian model comparison. Its last version stays readable and citable here.

Version 1 of 155 papers66 conceptsfrozen 2026-10-04

In traditional frequentist statistics, the $p$-value is a one-way street. It can tell you if your data is surprising under a null hypothesis, allowing you to "reject" that hypothesis. However, if the $p$-value is large, you are left in a statistical limbo: you have failed to reject the null, but you haven't proven it either. You cannot distinguish between a "null effect" (the phenomenon doesn't exist) and "insufficient data" (the experiment was too small to find it).

A growing body of research across psychology, neuroscience, and physics is moving toward a Bayesian framework to solve this "absence of evidence" problem. By using Bayes Factors and Relative Belief Ratios, researchers can now provide active evidence in favor of the null hypothesis, allowing for the formal "acceptance" of null effects and the direct comparison of competing scientific theories.

The Intuition: Likelihood Ratios as Evidence

The core mechanism of this research direction is the Bayes Factor (BF). Unlike a $p$-value, which only considers the null hypothesis ($H_0$), the Bayes Factor is a ratio that compares how well two different hypotheses predict the observed data.

As Karsten M. Borgwardt and Zoubin Ghahramani demonstrate in Bayesian two-sample tests, this approach provides a symmetric measure of evidence. If the data is much more likely under the alternative hypothesis ($H_1$) than the null ($H_0$), the BF is large, supporting $H_1$. If the data is more likely under the null, the BF is small (less than 1), providing explicit support for $H_0$.

This allows for a continuous scale of certainty. In cosmology, Roberto Trotta suggests benchmarks for these ratios: a 20:1 ratio represents "strong" evidence, while 150:1 is "decisive." This scale helps researchers like Chenyang Zhang and Guosheng Yin resolve discrepancies in clinical trials, where a non-significant $p$-value might mask a statistically plausible risk that a Bayes Factor can still detect.

Distinguishing Fairness from Insufficiency

One of the most powerful applications of this framework is in "proving a negative." In Detecting implicit biases of large language models with Bayesian hypothesis testing, Shijing Si and colleagues show that Bayes Factors can distinguish between a model that is truly "fair" and a test that simply lacks enough data to prove it is biased.

Frequentist tests often label a model as "fair" simply because they fail to find a significant difference. The Bayesian approach, however, requires the data to actively support the "fairness" hypothesis. This is echoed in the work of John Lurquin on ego-depletion, where Bayesian analysis provided robust evidence that the widely-believed "self-control drain" effect was actually non-existent, rather than just poorly measured.

The Relative Belief Ratio: From Prior to Posterior

While Bayes Factors compare two models, the Relative Belief Ratio focuses on how the data changes our mind about a specific parameter. As defined by Michael Evans and Forough Fazeli-Asl, this ratio is the posterior density divided by the prior density at a specific point.

If the ratio is greater than 1, our belief in that value has increased; if it is less than 1, the data has "debunked" that value. In An Evidential Route to Asymptotic Bayes Optimality under Sparsity, Qiaoyu Liang and Haohua Chen use this mechanism to show that simple Gaussian priors can be just as effective as complex "heavy-tailed" priors for detecting sparse signals, provided you use an evidential framework rather than just looking at the posterior alone.

Regions of Practical Equivalence (ROPE)

In many fields, we don't care if two things are exactly identical; we care if they are "close enough" to be interchangeable. This is formalized as the Region of Practical Equivalence (ROPE).

Piotr Woźnicki uses this to compare AI-generated radiology reports to human experts. By defining a ROPE, researchers can use Bayesian inference to calculate the probability that the difference between an AI and a human falls within a negligible range. Similarly, Jinyang Yu and Kaustubh R. Patil use Bayesian equivalence testing to show that different model validation techniques are practically equivalent, even if they aren't numerically identical.

Beyond Binary: Comparing Multiple Theories

Finally, this research direction moves science away from the "Null vs. Alternative" binary toward a more nuanced Model Comparison paradigm. Instead of asking "Is there an effect?", researchers like Zoltán Dienes and Piero Amodio ask "Which of these three theories best explains the data?".

In Testing two competing hypotheses for Eurasian jays’ caching, the authors formalized qualitative behavioral theories into quantitative models. By calculating the posterior probability for each, they could conclude that neither of the leading theories explained the birds' behavior as well as a simple preference model. This "suspension of judgment," as Alessio Benavoli and Giorgio Corani call it in their Bayesian Wilcoxon test, is a hallmark of the new Bayesian rigor: the ability to say the data is too ambiguous to support any theory, rather than being forced into a "significant" or "non-significant" box.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_quantifying_evidence_for_the_null_hypoth_1a88491b,
  title = {Quantifying Evidence for the Null Hypothesis with Bayes Factors},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quantifying-evidence-for-the-null-hypothesis-with-bayes-factors/rdr_1a88491b0aef4a158430f3ecc07c6d9b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_quantifying_evidence_for_the_null_hypoth_1a88491b_v1,
  title = {Quantifying Evidence for the Null Hypothesis with Bayes Factors},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quantifying-evidence-for-the-null-hypothesis-with-bayes-factors/rdr_1a88491b0aef4a158430f3ecc07c6d9b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}