Introduction
In contemporary machine learning (ML), model explainability is frequently treated as a definitive tool for scientific discovery. Researchers often train high-performing models—such as Random Forests or Deep Neural Networks—and subsequently apply techniques like SHAP (Shapley Additive exPlanations) or feature importance to identify "risk factors" or "key drivers." However, Matthew J. Vowels’ research, "Trying to Outrun Causality in Machine Learning," argues that this practice rests on a precarious foundation. The study demonstrates that ML models are fundamentally constrained by the underlying causal structure of the data; failing to account for this structure can lead explainability tools to misidentify critical variables as irrelevant.
The Problem: Prediction vs. Explanation
The primary issue lies in the divergence between predictive performance and causal explanation. ML models are optimized for prediction. When a model "conditions" on a variable (incorporates it as a predictor), it alters the statistical relationships among all other variables in the feature set. In complex data structures—characterized by mediators, colliders, or shared ancestors—the model may "block" the path of a genuine causal driver, rendering it seemingly unimportant in feature rankings. Vowels posits that increased algorithmic flexibility (e.g., deep learning) cannot circumvent these fundamental causal rules.
Methodology
The study evaluates several standard approaches against varying data structures:
- Linear Regression (LR): A baseline for standard statistical conditioning.
- Random Forests (RF): Assessed via impurity-based importance and SHAP values.
- MultiLayer Perceptrons (MLP): A neural network architecture evaluated using SHAP.
- Bivariate Mutual Information (MI): A non-linear measure of shared information between variable pairs, independent of other features.
- Causal Discovery (PC Algorithm): An algorithm designed to infer the underlying directed acyclic graph (DAG) from observational data.
Datasets and Causal Structures
The research utilizes two synthetic datasets with known "ground truth" causal graphs:
- The Trivial Dataset: Comprising eight independent predictors that directly cause the outcome, serving as a control.
- The Complex Dataset: A 9-variable system based on a Structural Causal Model (SCM) featuring mediators ($X \to D \to Y$), colliders ($C \to X \leftarrow A$), and backdoor paths (non-causal correlations from shared ancestors).
These structures illustrate how variables relate to one another as parents, children, ancestors, or descendants, forming the basis for the "blocking" effects observed in the results.
Key Insights: The "Blocking" Effect and Model Behavior
A central finding is the "Blocking" Effect. In the complex dataset, variable $X$ is a primary cause of outcome $Y$. However, if the model also includes variable $D$ (a mediator where $X \to D \to Y$), the model captures the necessary predictive information from $D$ and assigns $X$ near-zero importance. This highlights that explainability tools provide local insights into model utility rather than global truths about physical reality.
Furthermore, the study found that Random Forests exhibited unpredictable behavior. Unlike Linear Regression or Neural Networks, which reacted to causal structures in ways consistent with d-separation rules, RF importance rankings shifted inconsistently when variables were removed, making them particularly unreliable for scientific inference.
Results
- Trivial Case: All methods (LR, RF, MLP) successfully identified all variables as equally important.
- Complex Case: When all variables were included, LR and MLP/SHAP assigned nearly zero importance to variable $X$, a primary cause. Random Forest importance was highly volatile and unpredictable.
- Causal Success: Only Bivariate Mutual Information and the PC Algorithm consistently identified the true relevance of primary drivers like $X$, regardless of the presence of other variables.
Conclusion and Future Work
The study concludes that researchers seeking to understand causal mechanisms should not rely solely on ML explainability plots. Instead, the integration of Causal Discovery and the construction of Directed Acyclic Graphs (DAGs) should be a mandatory precursor to applying ML models.
Future research is suggested to explore these limitations in more advanced architectures like Transformers and to test them on messier, non-linear real-world datasets with categorical outcomes. The ultimate takeaway is clear: you cannot "deep learn" your way out of a blocked causal path.
Summary updated