Feature relevance quantification in explainable AI: A causal problem

Dominik Janzing, Lenon Minorics, Patrick BlöbaumAISTATS 2020 · OpenReview

Abstract

We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped features. We argue that the interventional (marginal) distribution is the correct choice for explaining model behavior, as it avoids "guilt by association" and adheres to the causal structure of computer algorithms.

Introduction

Explainable AI (XAI) aims to make complex machine learning models transparent by quantifying the contribution of individual features to a model's output. A central challenge in methods like SHAP (SHapley Additive exPlanations) is how to handle "missing" features during the attribution process. Since most models require a complete input vector, researchers must simulate the absence of a feature by replacing it with an expected value. This leads to a fundamental debate: should we use the conditional expectation (the observational approach) or the marginal expectation (the interventional approach)?

The Problem of "Guilt by Association"

A common argument for conditional expectations is that they produce "more realistic" feature combinations by respecting real-world correlations. However, this paper demonstrates that this approach leads to a phenomenon termed "guilt by association." If an irrelevant feature is highly correlated with a relevant one, the conditional approach will incorrectly assign it high importance.

Causal graph illustrating how a common cause Z can lead to correlation between features X, potentially causing "guilt by association" in observational attribution methods.

As shown in Figure 1, if a latent variable $Z$ acts as a common cause for features $X_1, X_2,$ and $X_3$, these features will be statistically dependent. If the model $Y$ only truly depends on $X_1$, a conditional attribution method might still credit $X_2$ and $X_3$ because they provide information about $X_1$. This violates the Sensitivity axiom, which requires that a feature with no impact on the model's output should receive zero attribution.

A Causal Framework for Attribution

The authors propose that feature attribution is inherently a causal problem. When we change an input to a function $f(x)$, we are performing a manual intervention, represented by Judea Pearl’s $do$-operator. In the causal graph of a computer program, the inputs are the direct causes of the output.

The interventional framework for feature attribution, where some features are fixed (intervened upon) while others are marginalized out to simulate their absence.

Figure 2 illustrates this interventional setup. To calculate the contribution of a subset of features, we fix their values (e.g., $x_1, x_2$) and integrate over the remaining features ($X_3, X_4, X_5$) using their marginal distribution. This "breaks" the real-world correlations, ensuring that the attribution reflects only the model's internal logic rather than external statistical dependencies. This approach is not only theoretically sound but also computationally cheaper than estimating complex conditional distributions.

Key Results and Insights

Using synthetic Gaussian data and real-world Human Activity Recognition (HAR) data, the authors compared the two approaches against "ground truth" importance (model coefficients). The results consistently showed:

  1. Sensitivity Compliance: The marginal approach correctly assigned zero importance to features with zero coefficients, whereas the conditional approach failed.
  2. Error Reduction: Quantitative analysis of error distributions showed that marginal expectations lead to attributions that are significantly closer to the true feature contributions.
  3. SHAP's "Happy Accident": While the original SHAP theory pointed toward conditional expectations, its popular implementation used marginal ones for efficiency. The authors argue that SHAP was actually "right for the wrong reasons," and that attempts to "fix" it by making it truly conditional actually degrade the quality of explanations.

Conclusion

The move toward conditional expectations in XAI, while intended to avoid "out-of-distribution" samples, fundamentally changes the objective from explaining the model to explaining the data's correlations. For true model explanation, the interventional approach using marginal distributions is the conceptually correct choice, providing a robust framework that respects the causal structure of algorithmic decision-making.

Summary updated

Abstract

We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped feat...
BibTeX
@inproceedings{janzing2020feature,
  author = {Dominik Janzing and Lenon Minorics and Patrick Blöbaum},
  title = {{Feature relevance quantification in explainable AI: A causal problem}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=POgkgch0G1},
}
Figure