When we ask an interpretability tool to explain a machine learning model's prediction, we are usually handed a map of feature importance scores. If a model predicts a patient has a high risk of heart disease, a tool like SHAP or LIME might point to their age or blood pressure as the primary drivers.
Intuitively, we interpret this to mean: "If we change this feature, the prediction will change." But in reality, standard interpretability tools only show us what the model found statistically associated with the outcome. They capture correlations, not causes. This disconnect is known as the causal attribution gap.
As Umang Bhatt et al. observe in Explainable Machine Learning in Deployment, this lack of causal grounding severely limits the utility of explanations in high-stakes domains. If an explanation highlights spurious correlations rather than the actual logic driving a decision, human operators cannot safely rely on it to make critical real-world interventions.
The Illusion of Feature Importance
Why do traditional interpretability methods fail to capture causality? The root of the problem lies in how these methods define "importance."
Most tools calculate a feature's importance based on its marginal contribution to a model's predictive accuracy. However, as Schwab et al. argue in CXPlain: Causal Explanations for Model Interpretation under Uncertainty, this approach only explains the model's internal statistical logic. It does not uncover the true physical or real-world causal mechanisms linking the input features to the actual environment. If a model exploits a shortcut—such as using the background texture of an image to classify an animal—traditional attribution tools will faithfully highlight the background as "important."
This problem isn't confined to post-hoc explanations; it also plagues mechanistic interpretability, which attempts to map internal neural representations to human-understandable concepts. Sparse Autoencoders (SAEs) are widely used to decompose the hidden states of language models into interpretable features. Yet, Menon et al. demonstrate in Analyzing (In)Abilities of SAEs via Formal Languages that standard SAE training methods frequently identify features that align perfectly with human concepts but possess zero causal influence over the model's actual output. These features are merely "along for the ride."
This happens because standard dictionary learning optimizes for statistical frequency and reconstruction fidelity rather than computational impact. As Olmo et al. explain in Features that Make a Difference: Leveraging Gradients for Improved Dictionary Learning, passive data representations often overshadow small-magnitude features that are mechanistically vital for the model's downstream computation.
The Intervention Trap
To prove that an internal feature or direction actually matters, researchers often turn to activation patching—manually swapping an internal activation from one context into another to see if the model's behavior changes as expected. While this feels like a rigorous causal intervention, it can create its own illusions.
In Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching, Makelov et al. reveal that end-to-end causal effects from patching can be highly deceptive. The act of intervening can introduce artificial distortions or out-of-distribution artifacts that trigger downstream behavioral shifts. Consequently, a researcher might conclude they have found a functionally relevant feature subspace, when in reality, the observed effect was merely an artifact of the intervention itself, completely disconnected from the model's natural internal mechanisms.
Model-Faithful vs. Data-Faithful Attribution
To resolve the causal attribution gap, we must clarify what we want our explanations to be faithful to.
Qiheng Sun formalizes this crucial distinction in Data-faithful Feature Attribution: Mitigating Unobservable Confounders via Instrumental Variables:
- Model-faithful attribution explains the internal logic of the predictive model, capturing how it navigates its inputs, regardless of whether those inputs are plagued by spurious correlations.
- Data-faithful attribution attempts to represent the true, real-world causal mechanisms of the underlying data-generating process.
When a model is trained on data with unobservable confounders, an explanation that is perfectly faithful to the model will be fundamentally misleading about the real world. For a clinician or policymaker, a model-faithful explanation might suggest intervening on a feature that has no real-world causal power over the outcome.
Furthermore, traditional attribution methods compress a feature's role into a single scalar score, masking how it operates causally. Esma Balkir addresses this in Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detection by splitting feature importance into two distinct causal dimensions:
- Necessity: Is this specific feature required for the model to maintain its prediction?
- Sufficiency: Is this feature alone enough to trigger the prediction, even if everything else changes?
By separating these roles, we can diagnose whether a model is relying on a feature as a mandatory logical step or as a shortcut trigger.
Passive Observation (Correlational XAI)
[ Input Features ] ───( Statistical Association )───> [ Model Prediction ]
│
└──> High Attribution Score (But is it a true cause?)
Active Probing (Causal XAI)
[ Intervention / Do-Calculus ] ───> [ Internal Pathway ] ───> [ Behavioral Shift ]
│
Verified Causal Driver <───────┘
Bridging the Gap with Causal Frameworks
To move past the limitations of correlational interpretability, researchers are actively integrating formal causal frameworks—such as structural causal models and Pearl's do-calculus—directly into the explanation process.
Instead of passively observing which features correlate with outputs, new methods enforce causal constraints during feature extraction. For instance, to ensure that extracted features are not just passive bystanders, Menon et al. introduce a Causal Loss function that penalizes autoencoders when their reconstructed features fail to actively drive the target model's behavior under intervention.
To achieve data-faithfulness in the presence of hidden confounders, Qiheng Sun introduces frameworks like IV-SHAP and IV-IG. By leveraging Instrumental Variables—variables that influence the treatment but are independent of the confounder—these methods isolate the true, unconfounded causal effect of features, ensuring explanations remain faithful to the real-world data-generation process.
Similar advancements are occurring in structured domains. In Generative Causal Explanations for Graph Neural Networks, Lin et al. utilize the framework of Granger causality to train a generative model. Instead of highlighting nodes that happen to be present, their framework systematically identifies the exact subgraphs that causally dictate a Graph Neural Network's predictions.
Ultimately, these approaches aim to establish a rigorous mathematical foundation for model explanations. As Cheng et al. demonstrate in Linking Model Intervention to Causal Interpretation in Model Explanation, it is possible to define the exact theoretical conditions under which intuitive "what-if" model perturbations successfully align with real-world direct causes. By anchoring explainability in the language of causality, the field is moving away from merely describing what a model looks at, and toward uncovering why it makes a decision.
Key Foundational Work
Predicting and visualizing psychological attributions with a deep neural network (2016) — contributed 1 concept elements to this direction.