Probabilities of Causation: Role of Observational Data

Ang Li, Judea PearlAISTATS 2023

Introduction

In causal inference, a fundamental question is the "Probability of Necessity and Sufficiency" (PNS). This metric asks: "What is the probability that an outcome occurred because of a specific action, and would not have occurred otherwise?" Because we cannot observe the same individual in two counterfactual states simultaneously, PNS is typically unidentifiable. Instead, researchers rely on bounding techniques that combine experimental data (from Randomized Controlled Trials) with observational data.

However, observational studies are often resource-intensive. This paper by Ang Li and Judea Pearl introduces a decision-making framework to determine whether the potential gain in narrowing these causal bounds justifies the cost of conducting an observational study.

Methodology: Quantifying Expected Improvement

The authors build upon the Tian-Pearl Theorem, which defines the sharpest possible bounds for PNS. The core innovation is the concept of Expected Improvement (EI). Since observational data is unknown prior to a study, the authors assume that observational distributions are uniformly distributed across their mathematically feasible intervals.

Using this assumption, they derive closed-form solutions to calculate the expected increase in the lower bound ($E(L' - L)$) and the expected decrease in the upper bound ($E(U - U')$).

Figure 1: The expected decrease in the upper bound as a function of experimental probabilities.

Key Insights: The "Sweet Spot" of Uncertainty

A primary finding of the research is that the value of observational data is not constant. The expected improvement is maximized when the initial experimental probabilities ($P(y_x)$ and $P(y_{x'})$) are both near 0.5.

As shown in Figure 1, if the experimental data already indicates a very strong effect (near 1) or a very weak one (near 0), the bounds are already relatively tight or the outcome is highly predictable, making additional observational data less impactful. Conversely, when uncertainty is high (near 0.5), observational data provides the most significant sharpening of the causal bounds.

Figure 3: 2D slices showing the expected increase in the lower bound for various fixed values of experimental data.

Figure 3 further illustrates this behavior for the lower bound. The peaks occur when $P(y_x)$ aligns with the uncertainty in the experimental setup, demonstrating how researchers can predict the utility of a study before it begins.

Practical Applications and Results

The paper validates these theorems through simulated case studies:

  • Vaccine Effectiveness: In a simulation of a pharmaceutical trial, the theorem predicted a high expected improvement (~0.22). Following through with an observational study narrowed the PNS range from a vague [0.05, 0.52] to a much more actionable [0.33, 0.41].
  • Marketing Strategy: In a car dealership "enticement" scenario, the theorem predicted a negligible improvement (0.01). This correctly signaled that tracking customer behavior would be a waste of resources, as the resulting bounds would remain too wide to inform strategy.

Conclusion and Future Directions

This framework allows researchers to put a "price" on information. By providing simple, closed-form formulas based on existing experimental results, it prevents the waste of resources on studies unlikely to yield definitive causal insights. While the current model relies on a uniform distribution assumption for unknown data, future work could incorporate more sophisticated priors or extend the logic to non-binary variables.

Summary updated

BibTeX
@inproceedings{li2023probabilities,
  author = {Ang Li and Judea Pearl},
  title = {{Probabilities of Causation: Role of Observational Data}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
}
Figure