Trying to Outrun Causality with Machine Learning: Limitations of Model Explainability Techniques for Identifying Predictive Variables

Matthew J. VowelsarXiv 2022 · arXiv

Introduction

In contemporary machine learning (ML), model explainability is frequently treated as a definitive tool for scientific discovery. Researchers often train high-performing models—such as Random Forests or Deep Neural Networks—and subsequently apply techniques like SHAP (Shapley Additive exPlanations) or feature importance to identify "risk factors" or "key drivers." However, Matthew J. Vowels’ research, "Trying to Outrun Causality in Machine Learning," argues that this practice rests on a precarious foundation. The study demonstrates that ML models are fundamentally constrained by the underlying causal structure of the data; failing to account for this structure can lead explainability tools to misidentify critical variables as irrelevant.

The Problem: Prediction vs. Explanation

The primary issue lies in the divergence between predictive performance and causal explanation. ML models are optimized for prediction. When a model "conditions" on a variable (incorporates it as a predictor), it alters the statistical relationships among all other variables in the feature set. In complex data structures—characterized by mediators, colliders, or shared ancestors—the model may "block" the path of a genuine causal driver, rendering it seemingly unimportant in feature rankings. Vowels posits that increased algorithmic flexibility (e.g., deep learning) cannot circumvent these fundamental causal rules.

Methodology

The study evaluates several standard approaches against varying data structures:

  • Linear Regression (LR): A baseline for standard statistical conditioning.
  • Random Forests (RF): Assessed via impurity-based importance and SHAP values.
  • MultiLayer Perceptrons (MLP): A neural network architecture evaluated using SHAP.
  • Bivariate Mutual Information (MI): A non-linear measure of shared information between variable pairs, independent of other features.
  • Causal Discovery (PC Algorithm): An algorithm designed to infer the underlying directed acyclic graph (DAG) from observational data.

Datasets and Causal Structures

The research utilizes two synthetic datasets with known "ground truth" causal graphs:

  1. The Trivial Dataset: Comprising eight independent predictors that directly cause the outcome, serving as a control.
  2. The Complex Dataset: A 9-variable system based on a Structural Causal Model (SCM) featuring mediators ($X \to D \to Y$), colliders ($C \to X \leftarrow A$), and backdoor paths (non-causal correlations from shared ancestors).
Basic causal graph structures including parent-child relationships, ancestors, and cycles.

These structures illustrate how variables relate to one another as parents, children, ancestors, or descendants, forming the basis for the "blocking" effects observed in the results.

Key Insights: The "Blocking" Effect and Model Behavior

A central finding is the "Blocking" Effect. In the complex dataset, variable $X$ is a primary cause of outcome $Y$. However, if the model also includes variable $D$ (a mediator where $X \to D \to Y$), the model captures the necessary predictive information from $D$ and assigns $X$ near-zero importance. This highlights that explainability tools provide local insights into model utility rather than global truths about physical reality.

Illustration of directed paths and the impact of adding edges on graph cycles.

Furthermore, the study found that Random Forests exhibited unpredictable behavior. Unlike Linear Regression or Neural Networks, which reacted to causal structures in ways consistent with d-separation rules, RF importance rankings shifted inconsistently when variables were removed, making them particularly unreliable for scientific inference.

Results

  • Trivial Case: All methods (LR, RF, MLP) successfully identified all variables as equally important.
  • Complex Case: When all variables were included, LR and MLP/SHAP assigned nearly zero importance to variable $X$, a primary cause. Random Forest importance was highly volatile and unpredictable.
  • Causal Success: Only Bivariate Mutual Information and the PC Algorithm consistently identified the true relevance of primary drivers like $X$, regardless of the presence of other variables.

Conclusion and Future Work

The study concludes that researchers seeking to understand causal mechanisms should not rely solely on ML explainability plots. Instead, the integration of Causal Discovery and the construction of Directed Acyclic Graphs (DAGs) should be a mandatory precursor to applying ML models.

Future research is suggested to explore these limitations in more advanced architectures like Transformers and to test them on messier, non-linear real-world datasets with categorical outcomes. The ultimate takeaway is clear: you cannot "deep learn" your way out of a blocked causal path.

Summary updated

Abstract

Machine Learning explainability techniques have been proposed as a means of `explaining' or interrogating a model in order to understand why a particular decision or prediction has been made. Such an ability is especially important at a time when machine learning is being used to automate decision processes which concern sensitive factors and legal outcomes. Indeed, it is even a requirement according to EU law. Furthermore, researchers concerned with imposing overly restrictive functional form (e.g., as would be the case in a linear regression) may be motivated to use machine learning algorithms in conjunction with explainability techniques, as part of exploratory research, with the goal of identifying important variables which are associated with an outcome of interest. For example, epidemiologists might be interested in identifying `risk factors' - i.e. factors which affect recovery from disease - by using random forests and assessing variable relevance using importance measures. However, and as we demonstrate, machine learning algorithms are not as flexible as they might seem, and are instead incredibly sensitive to the underling causal structure in the data. The consequences of this are that predictors which are, in fact, critical to a causal system and highly correlated with the outcome, may nonetheless be deemed by explainability techniques to be unrelated/unimportant/unpredictive of the outcome. Rather than this being a limitation of explainability techniques per se, we show that it is rather a consequence of the mathematical implications of regression, and the interaction of these implications with the associated conditional independencies of the underlying causal structure. We provide some alternative recommendations for researchers wanting to explore the data for important variables.
BibTeX
@misc{vowels2022trying,
  author = {Matthew J. Vowels},
  title = {{Trying to Outrun Causality with Machine Learning: Limitations of Model Explainability Techniques for Identifying Predictive Variables}},
  year = {2022},
  eprint = {2202.09875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.09875},
}
Figure