In empirical research, discovering a mechanism that connects a cause to an effect is a major milestone. Scientists frequently use mediation analysis to uncover these bridges: if an exposure ($X$) causes an outcome ($Y$), we look for an intervening variable, a mediator ($M$), through which the causal effect flows.
When a statistical test confirms that $M$ is a significant mediator, it feels like a victory—we have peeked inside the "black box" of causality. However, a cross-disciplinary pattern of evidence reveals a critical blind spot in this approach: the partial mediation gap. Even when a mediator is highly statistically significant, it frequently explains only a small fraction of the total causal effect. The remaining, unexplained variance means that our understanding of the underlying mechanism remains fundamentally incomplete.
The Anatomy of a Partial Mediator
To understand why this gap matters, we must look at how mediation is quantified. The total causal effect of $X$ on $Y$ is the sum of the direct effect (the path from $X \to Y$ bypassing $M$) and the indirect effect (the path $X \to M \to Y$). When researchers state that a mediator explains 10% of an effect, they are revealing that 90% of the causal force is either direct or traveling through alternative, unmeasured pathways.
This discrepancy spans across entirely different fields of study, from genetics to behavioral science:
- Genetics and Biomedicine: Using Mendelian Randomization (MR) to establish robust causal links, Ruiling Lu et al. demonstrated in Causal relationship between asthma and ulcerative colitis and the mediating role of interleukin-18: a bidirectional Mendelian study and mediation analysis that adult-onset asthma causally increases the risk of ulcerative colitis. However, the inflammatory cytokine Interleukin-18 (IL-18) acts as only a minor mediator. The authors note that this small fraction implies the gut-lung axis involves a highly complex network far beyond simple cytokine signaling. Similarly, Guangyang Ou et al. found that while lipid-lowering drugs reduce atrial fibrillation risk, the identified inflammatory mediator (FGF5) accounts for less than 10% of the protective effect in Lipid-lowering drugs, circulating inflammatory factors, and atrial fibrillation: a mediation Mendelian randomization study, leaving alternative pathways like structural or metabolic shifts completely unaccounted for.
- Psychology and Sociology: In The relationship between uncertainty and acute procedure anxiety among surgical patients in Chinese mainland: the mediating role of resilience, Suwan Dai et al. found that psychological resilience accounted for roughly 31% of the total effect of illness uncertainty on acute procedure anxiety. While valuable, this leaves nearly 70% of the variance unexplained. When examining adolescent self-injury in The mediating role of cognitive flexibility in the relationship between social support and non-suicidal self-injury among adolescents: a multicenter cross-sectional study, Chenchen Wang identified cognitive flexibility as a mediator for the protective effects of social support, but the small proportion of variance explained forced the conclusion that a complex interplay of other factors (like impulse control or emotional regulation) must be at play.
- Artificial Intelligence: Even in artificial systems, partial mediation appears. Investigating how large language models process moral framing, Phongsakon Mark Konrad et al. isolated a specific routing pathway introduced by instruction tuning in Decoded but Unused: Instruction Tuning Routes Moral Framing into the Judgment Readout. Yet, this pathway only explained a portion of the overall framing effect, leaving the complete circuit map elusive.
Why the Gap Matters for Interventions
Focusing exclusively on a statistically significant but minor mediator can lead to weak or ineffective interventions. If a mediator explains only a fraction of the variance, designing a solution around it ignores the vast majority of the causal engine.
For example, Qiang Xu et al. utilized MR in Identifying the mediating role of socioeconomic status on the relationship between schizophrenia and major depressive disorder: a Mendelian randomisation analysis to show that socioeconomic status accounts for only about 10% of the causal link between schizophrenia and major depressive disorder. If clinicians or policymakers treat social deprivation as the primary driver of this comorbidity, their interventions will leave 90% of the risk unaddressed, pointing to a massive need to find the remaining biological pleiotropy or unmeasured environmental mediators.
Similarly, in prenatal development, Xiao Song Dong et al. demonstrated in Adverse events during pregnancy, circulating metabolites, and congenital malformations: a Mendelian randomization study that metabolic mediators like glycine explain only a tiny percentage of the causal link between pregnancy complications and birth defects. Treating glycine levels alone would fail to safeguard fetal health against the remaining, unaccounted mechanical or hormonal stressors.
| Study | Exposure ($X$) | Outcome ($Y$) | Identified Mediator ($M$) | The Unexplained Gap |
|---|---|---|---|---|
| Lu et al. | Asthma | Ulcerative Colitis | Interleukin-18 (IL-18) | High (Majority driven by unknown gut-lung pathways) |
| Xu et al. | Schizophrenia | Major Depressive Disorder | Socioeconomic Status | ~90% (Likely biological pleiotropy) |
| Dai et al. | Illness Uncertainty | Procedure Anxiety | Psychological Resilience | ~69% (Missing social/personality factors) |
| Li | Occupational Status | Health Outcomes | Work Intensity, Income, Hours | ~40% (Unmeasured lifestyle/environmental factors) |
Moving Beyond Single-Mediator Models
The persistence of unexplained mediation variance is forcing a paradigm shift. Instead of celebrating partial mediation as a final answer, contemporary researchers treat the unexplained variance as a quantitative metric of our ignorance—and a roadmap for what to discover next.
To bridge this gap, the research direction is moving away from simple, linear $X \to M \to Y$ models and toward two robust strategies:
- Multi-Factorial Network Integration: Acknowledging that single biomarkers or psychological traits are insufficient, researchers are building parallel and chaining mediation models. For instance, single-author work by Yingji Li in Occupational status and health disparities among workers—An empirical study based on China health and nutrition survey data explicitly combined work intensity, income, and hours to account for 60% of the effect between occupation and health, demonstrating that grouping multiple distinct pathways is necessary to shrink the residual variance.
- AI-Driven Causal Discovery: When the missing mediators are completely unknown, traditional hypothesis testing fails. In Toward a causal model of chronic back pain: Challenges and opportunities, J. Russell Huie et al. point out that current psychological and social factors only account for 20% to 33% of the relationship between chronic back pain and disability. To uncover the missing 67% to 80% of the mechanism, the authors advocate for abandoning simple correlational analysis in favor of running AI-driven causal discovery algorithms (such as the PC, FCI, and FGES algorithms) on massive, multi-modal datasets.
Ultimately, the partial mediation gap teaches us that statistical significance is not mechanical completeness. By systematically measuring and targeting the unexplained variance, modern research is turning unexplained gaps into new frontiers for causal discovery.
Key Foundational Work
Causal relationship between asthma and ulcerative colitis and the mediating role of interleukin-18: a bidirectional Mendelian study and mediation analysis (2023) — contributed 3 concept elements to this direction.
Identifying the mediating role of socioeconomic status on the relationship between schizophrenia and major depressive disorder: a Mendelian randomisation analysis (2023) — contributed 1 concept elements to this direction.
The relationship between uncertainty and acute procedure anxiety among surgical patients in Chinese mainland: the mediating role of resilience (2023) — contributed 1 concept elements to this direction.