Abstract
Simpson's paradox is an obstacle to establishing a probabilistic association between two events $a_1$ and $a_2$, given a third (lurking) random variable $B$. This research focuses on scenarios where the random variables $A$ (combining $a_1, a_2$, and their complements) and $B$ share a common cause $C$, which may be unobserved. We demonstrate that for minimal cases—binary or scalar Gaussian variables—the assumption of a common cause $C$ directs the choice toward the fine-grained, conditional data over aggregated results. Specifically, conditioning over any binary common cause $C$ establishes the same direction of association as conditioning over $B$ in the original formulation. This suggests that for the minimal common cause, one should choose the option of Simpson's paradox that assumes conditioning over $B$ and not its marginalization.
Introduction
Simpson’s paradox occurs when a statistical trend observed in multiple groups disappears or reverses when the groups are combined. This phenomenon poses a significant challenge in fields ranging from epidemiology to artificial intelligence: when faced with conflicting trends at different levels of aggregation, which data should guide decision-making? Traditional probabilistic approaches often lack the criteria to resolve this without specific causal knowledge. This paper proposes a resolution by invoking the "common cause principle."
The Common Cause Principle
The authors move beyond standard probability by introducing a "lurking" or "screening" variable $C$ that acts as a common cause for both the treatment/effect variables ($A$) and the confounding side variable ($B$). In this framework, the correct association between $a_1$ and $a_2$ is defined via conditioning over $C$. This setup generalizes the original Simpson's paradox, where the two contradicting options refer to two particular and different causes $C$.
Theoretical Resolutions
The paper provides mathematical proofs for "minimal" scenarios, which are the most frequent in practice:
- The Binary Case (Theorem 1): If $A, B,$ and $C$ are all binary, conditioning on any binary common cause $C$ yields the same direction of association as conditioning on $B$. This implies that if a common cause is suspected, the fine-grained (conditional) data is the mathematically consistent choice.
- The Gaussian Case (Theorem 2): Extending this logic to continuous variables, the authors prove that in a minimal setup of three scalar Gaussian variables, the conditional option is the mathematically consistent choice.
Using non-informative Dirichlet priors, the authors estimate that Simpson's paradox occurs in approximately 4.29% of random three-variable binary cases, suggesting it is a generic feature of probabilistic data rather than a rare fluke.
Empirical Validation
The framework was applied to real-world examples to demonstrate its utility:
- Smoking and Survival: In a 20-year study of women, aggregated data suggested smokers lived longer. However, age-adjusted data showed the opposite. By treating genetics as a potential common cause $C$, the model correctly aligns with the age-adjusted (fine-grained) conclusion.
- COVID-19 Fatality Rates: Comparing Italy and China (2020), aggregated data suggested higher fatality in Italy, while age-specific groups suggested China had higher rates. The common cause principle suggests that "healthcare system quality" could be the latent variable $C$, favoring the age-stratified data.
Conclusion
The research demonstrates that for the most common formulations of Simpson's paradox, one does not need to measure the latent common cause to determine the correct course of action. The mere assumption of its existence in a minimal setup dictates that conditioning over the confounding variable $B$ is the correct approach. This provides a robust theoretical foundation for preferring fine-grained data in decision-making processes.
Summary updated