An Empirical Study of the Simplest Causal Prediction Algorithm

Jerome Cremers, Joris M. MooijACI @ UAI 2015 · OpenReview

TL;DR

This paper investigates a simple causal prediction algorithm that relies on identifying specific patterns of conditional independences among four variables. While sound and consistent in theory, the study reveals that this method's prediction accuracy degrades significantly with an increasing number of latent variables, even failing to outperform simple non-causal baselines in simulations.

What Problem Does This Paper Address?

Causal inference aims to predict the outcomes of interventions (e.g., "what happens if we artificially reduce gene X's expression?"). This is distinct from standard prediction tasks that only use observational data. A major challenge is to make these causal predictions using only observational data, without knowing the underlying causal structure of the system. Existing methods often rely on learning the entire causal graph, which can be statistically challenging and prone to errors propagating from conditional independence tests. This paper explores a simpler approach that focuses on identifying specific, minimal patterns of conditional independences to make causal predictions, aiming for reliability over completeness.

Methods

The core of the paper revolves around a specific causal prediction algorithm that identifies "Extended Y-structures." These structures are defined by a particular set of four conditional independence relations among four variables (X, Y, Z, U). The algorithm essentially searches for these patterns in observational data. If such a pattern is found, the algorithm concludes that the interventional distribution of Y given X is the same as the observational distribution, i.e., $p(Y | do(X=x)) = p(Y | X=x)$. This conclusion is derived from established causal discovery rules (Claassen and Heskes, 2011) and a causal prediction rule (Entner et al., 2013). The paper also introduces Algorithm 2, a more general pattern search framework that can be used to find these Y-structures and other patterns.

Datasets

The paper uses simulated data for its experiments. They generate random causal Directed Acyclic Graphs (DAGs) with varying numbers of variables (p = 10, 30, 50, 70). For each DAG, they generate observational data and simulate interventions by setting a target variable to a constant value. No new benchmark datasets are introduced.

Key Insights and Ideas

  • Simplicity vs. Robustness Trade-off: The paper highlights that while focusing on simple, identifiable patterns (like Y-structures) can be appealing for reliability, it comes at the cost of predictive accuracy, especially in the presence of unobserved confounders.
  • Faithfulness Violations are Costly: The study strongly suggests that violations of the "faithfulness" assumption (where conditional independences in the data perfectly reflect independences in the causal graph) become increasingly likely with more latent variables. These violations can severely degrade the performance of constraint-based methods.
  • Latent Variables are the Achilles' Heel: The most striking finding is how quickly the algorithm's performance deteriorates as the number of latent variables (or simply variables in the system, as they simulate causally sufficient DAGs) increases. Even with a moderate number of variables (p=50) and a substantial sample size (N=3000), the simple causal prediction method fails to outperform inconsistent non-causal baselines.
  • "Error Correction" is Not Guaranteed: The authors observe that errors in individual independence tests don't always cancel out; sometimes, combining tests can amplify errors, and other times, it seems to lead to "error correction," but the reasons are unclear.

Key Results

  • Discovery of Y-structures: The pattern search algorithm (Algorithm 2) generally outperforms random guessing in detecting Y-structures and Extended Y-structures, achieving higher precision and recall on simulated data.
  • Causal Prediction Accuracy: For causal prediction tasks (estimating $p(Y | do(X=x))$), the simple algorithm shows promise for small numbers of variables (p=10), outperforming simple baselines.
  • Performance Degradation: For p=50 variables, the causal prediction error of the proposed method becomes worse than two simple baselines: one that always predicts no change ($p(Y|do(X=x)) = p(Y)$) and one that equates causation with correlation ($p(Y|do(X=x)) = p(Y|X=x)$). This indicates that the method's predictions are unreliable in higher dimensions.
  • Precision vs. Recall Trade-off: As more tests are combined to define more complex patterns (like Y1 and Y2), recall generally decreases, while precision can sometimes increase or decrease depending on the pattern.

Limitations and Future Work

  • Uniform Consistency: The paper acknowledges that the algorithm is not uniformly consistent, meaning there's no guarantee on how many samples are needed to be confident about a prediction.
  • Faithfulness Assumption: The method relies heavily on the faithfulness assumption, which is shown to be problematic in the presence of many latent variables.
  • Linear-Gaussian Assumption: The simulations are conducted under a linear-Gaussian model. While this simplifies independence testing, it might not generalize to all types of data distributions.
  • Future Work: The authors suggest exploring stronger assumptions like "strong faithfulness" for linear-Gaussian cases to achieve uniform consistency. They also point to the need to understand why certain patterns lead to reliable predictions while others do not, especially in high-dimensional settings. Investigating the impact of latent variables and their interaction with faithfulness violations remains a key area for future research.

Summary updated

BibTeX
@inproceedings{cremers2015empirical,
  author = {Jerome Cremers and Joris M. Mooij},
  title = {{An Empirical Study of the Simplest Causal Prediction Algorithm}},
  booktitle = {ACI @ UAI},
  year = {2015},
  url = {https://openreview.net/forum?id=sGa1_EdkrL},
}
Figure