Frequentist Inference and the Misinterpretation of P-Values

25 concepts11 papers2 proposals

In scientific research, the $p$-value is often treated as a binary gatekeeper: a value below 0.05 grants entry into the realm of "truth," while a value above it consigns a study to the scrap heap of failed experiments. However, as Ming Wang and Oliver Y. Chén et al. argue, this rigid reliance on an arbitrary threshold obscures the actual meaning of the metric, contributing significantly to the modern reproducibility crisis.

The fundamental problem is not the $p$-value itself, but a pervasive misunderstanding of what it measures. To understand why this happens, we must look at the logical gap between what researchers want to know and what frequentist statistics actually provides.

The Fallacy of the Transposed Conditional

Most researchers want to know the probability that their hypothesis is correct given the data they collected—mathematically, $P(H|Data)$. However, a $p$-value provides the inverse: the probability of observing the data (or something more extreme) assuming the null hypothesis is true, or $P(Data|H_0)$.

Maurits Clemens Kaptein et al. describe the confusion of these two concepts as the fallacy of the transposed conditional. It is a logical error similar to the "Prosecutor’s Fallacy" in law, where, as A. Philip Dawid explains, the rarity of a forensic match (the probability of the evidence given innocence) is incorrectly equated with the probability of innocence itself.

In a research context, Daniel Khashabi et al. found a significant gap between researchers' confidence in their statistical knowledge and their actual ability to define these metrics. This gap leads many to believe that a low $p$-value directly quantifies the likelihood that their research hypothesis is true, when it actually only measures how "surprising" the data is under the assumption that no effect exists.

The Trend Fallacy and the Proof of the Null

When a $p$-value fails to cross the 0.05 threshold, researchers often fall into two opposite but equally problematic traps:

  1. The Trend Fallacy: Zhichao Jin et al. highlight a common practice where researchers claim a "trend toward significance" for results like $p = 0.08$. This suggests that the data is "almost" meaningful. In frequentist terms, this is a misconception; if the alpha threshold is not met, the result is simply non-significant. Attempting to find patterns in inconclusive data risks overstating the importance of what might be mere noise.
  2. Proof of the Null: Conversely, Ming Wang emphasizes that a $p$-value greater than 0.05 does not prove that the null hypothesis is true. It merely indicates that the evidence was insufficient to reject it. As Zhengxiao Wei et al. point out, traditional null hypothesis significance testing (NHST) is fundamentally incapable of quantifying evidence for the null; it can only provide evidence against it.

Inconsistency and the Need for Context

The misuse of $p$-values often manifests as internal contradictions in research papers. Xianqiang Liu notes that researchers sometimes report a "significant" $p$-value even when the associated confidence interval crosses the null value (e.g., a hazard ratio of 1.0). Such inconsistencies undermine the credibility of the findings and suggest that the $p$-value is being used as a ritualistic label rather than a coherent part of the evidence.

Furthermore, a low $p$-value does not indicate the magnitude or importance of an effect. A tiny, clinically irrelevant difference can be "statistically significant" if the sample size is large enough. This is why Lucio De Capitani et al. argue that $p$-values fail to quantify the "reproducibility probability" or the stability of a result.

Moving Toward Bayesian Intuition

To resolve these issues, many researchers advocate for a shift toward Bayesian frameworks. Unlike $p$-values, Bayesian methods like Bayes Factors and Posterior Intervals allow researchers to directly quantify the evidence for one hypothesis over another. Daniel Khashabi et al. suggest that these tools help move beyond binary "significant vs. non-significant" thinking.

Interestingly, the gap between the two worlds may not be as wide as it seems. Haolun Shi demonstrates that under specific conditions, $p$-values can be mathematically reconciled with Bayesian posterior probabilities. However, as Ognjen Arandjelovic cautions, simply refining how we use $p$-values may not be enough if the underlying assumptions remain unexamined. The path forward requires not just better math, but a more nuanced understanding of what it means to have "evidence" in science.

Key Foundational Work

A Retrospective Survey of Research Design and Statistical Analyses in Selected Chinese Medical Journals in 1998 and 2008 (2010) — contributed 3 concept elements to this direction.
Probability and Statistics in the Law (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_p_value_misinterpretation_b298bf2e,
  title = {p-value misinterpretation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/p-value-misinterpretation/rdr_b298bf2e910d486cbfaa51fa83324ac4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_p_value_misinterpretation_b298bf2e_v1,
  title = {p-value misinterpretation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/p-value-misinterpretation/rdr_b298bf2e910d486cbfaa51fa83324ac4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake