When we ask an explainable AI (XAI) method to explain a model's prediction, we are typically looking for the features that drove the decision. If an algorithm assigns a high importance score to a patient's zip code when predicting heart disease, we might assume the model has found a meaningful proxy for environmental health risks.
However, a growing body of research reveals a fundamental flaw in standard feature attribution methods: they routinely conflate statistical correlation with functional or causal relevance. This vulnerability leads to importance smearing or guilt by association, where completely irrelevant distractor features receive high importance scores simply because they are correlated with the true predictive features in the training data.
To fix this, researchers are shifting feature attribution from an associational framework to a causal one. By treating neural networks as structural causal models and utilizing physical or mathematical interventions, we can isolate a model's true internal logic from the background correlations of the data it was trained on.
Why Standard Explanations Mislead: The Conditional Trap
To understand why standard explainers fail, we must look at how they simulate the "absence" of a feature. Popular methods like SHAP or LIME calculate a feature's importance by comparing the model's prediction with the full input against its prediction when the feature is missing. But machine learning models cannot easily ingest "missing" data; they require a concrete value.
To bypass this, traditional methods use conditional expectations. If we want to remove a feature $X_i$, we look at the remaining features $X_S$ and sample $X_i$ from the conditional distribution $P(X_i | X_S)$.
As Najmi demonstrates in The Many Shapley Values for Model Explanation, this approach inadvertently blends the model's internal logic with the dataset's statistics. Imagine a model trained to predict house prices using two features: Number of Rooms (which the model heavily relies on) and Luxury Doorknobs (which the model completely ignores). In the real world, luxury doorknobs only appear in large mansions.
When a conditional explainer evaluates the model without the Luxury Doorknobs feature, it asks: "Given that this house has 10 rooms, what is the expected prediction when we average over the typical doorknobs found in 10-room houses?" Because the data correlates the two, the explainer never actually tests the model on a large house without luxury doorknobs. As a result, the non-predictive feature inherits a high importance score. Taufiq et al. refer to this phenomenon as importance smearing, where the true model logic is masked by feature co-variation.
This failure mode is widespread. Tan et al. show that this "explanatory inversion" occurs across tabular, text, and image modalities, meaning standard explainers frequently hallucinate justifications based on coincidental data artifacts rather than the model's actual reasoning.
The Causal Remedy: The Do-Operator
To sever the link between data correlation and model logic, Janzing et al. proposed framing feature relevance strictly as a causal problem. Instead of conditioning on observed data, explainers should use Judea Pearl’s do-operation ($do(X_i)$), which breaks the arrows feeding into a variable in a causal graph.
In practice, applying a $do$-intervention means that when we remove a feature, we replace it with values sampled from its marginal distribution $P(X_i)$, completely ignoring the values of the other features.
$$\text{Conditional (Associational): } \mathbb{E}[f(X) | X_S]$$ $$\text{Interventional (Causal): } \mathbb{E}[f(X) | do(X_S)]$$
By breaking the statistical dependencies between inputs, we force the explainer to evaluate the model on counterfactual scenarios where correlations no longer hold. Returning to our housing example, an interventional explainer would test the model on a 10-room house forced to have cheap doorknobs. Because the model's output remains unchanged during this intervention, the explainer correctly assigns zero importance to the doorknobs.
This foundational shift has inspired several frameworks:
- Average Causal Effect (ACE): Chattopadhyay et al. treat the entire neural network as a Structural Causal Model (SCM), measuring the direct causal impact of input features on the output layer.
- do-Shapley: Jung et al. integrate $do$-calculus with game-theoretic Shapley values, preventing credit-splitting between highly correlated variables.
- Path Interventions: Moving beyond the input layer, Ahmad et al. perform interventions on internal neural pathways to map stable, faithful explanatory graphs of a model's reasoning.
Confounders and the Collider Trap
The necessity of a causal framework becomes even more pronounced when data dependencies are complex, such as when hidden variables or specific network structures introduce bias.
Unobservable Confounders
When an unobserved variable influences both an input feature and the target outcome, standard predictive models use the observed feature as a shortcut proxy for the hidden factor. As Sun details in Data-faithful Feature Attribution: Mitigating Unobservable Confounders via Instrumental Variables, standard attribution methods will incorrectly credit the observed feature for effects driven entirely by the invisible confounder. By leveraging Instrumental Variables (as seen in IV-SHAP), researchers can isolate the confounder-free variation in a feature, ensuring explanations reflect true underlying mechanisms rather than model shortcuts.
The Collider Trap
A counter-intuitive failure mode occurs when data contains a collider—a variable that is a common effect of two independent causes. Harel et al. mathematically proved that traditional feature importance metrics are highly susceptible to the collider trap. When an explanation model conditions on a collider, it creates a spurious statistical association between the independent causes. This causes the explainer to highlight variables that have zero causal effect on the target outcome, rendering the explanation logically nonsensical.
Balancing Causal Purity with Data Realism
While pure interventional methods ($do$-calculus) solve the problem of spurious correlation, they introduce a new challenge: the off-manifold problem.
When we break feature correlations by forcing a marginal distribution, we might generate highly unrealistic feature combinations—such as a house with 0 rooms but 5 bathrooms. Because the neural network was never exposed to such examples during training, its behavior on these "off-manifold" points can be highly unpredictable and unstable, as noted by Lakkaraju et al..
To resolve this tension, recent work seeks a middle ground. For instance, Taufiq et al. introduced ManifoldShap, a method that restricts causal interventions to the data manifold. By evaluating the model only on physically plausible counterfactuals while maintaining an interventional perspective, it prevents the model from behaving erratically on impossible inputs while still successfully distinguishing true causal drivers from mere statistical symptoms.
Key Foundational Work
Neural Network Attributions: A Causal Perspective (2019) — contributed 1 concept elements to this direction.
Robust Attribution Regularization. (2019) — contributed 1 concept elements to this direction.
Does the Phonology of L1 Show Up in L2 Texts? (2014) — contributed 1 concept elements to this direction.