Do Not Trust Additive Explanations

Alicja Gosiewska, Przemysław Biecek

Abstract

Explainable Artificial Intelligence (XAI) has received a great deal of attention recently. Explainability is being presented as a remedy for the distrust of complex and opaque models. Model agnostic methods such as LIME, SHAP, or Break Down promise instance-level interpretability for any complex machine learning model. But how faithful are these additive explanations? Can we rely on additive explanations for non-additive models? In this paper, we (1) examine the behavior of the most popular instance-level explanations under the presence of interactions, (2) introduce a new method that detects interactions for instance-level explanations, (3) perform a large scale benchmark to see how frequently additive explanations may be misleading.

Introduction

As machine learning models become increasingly complex, the demand for interpretability has led to the widespread adoption of local, model-agnostic explanation tools. Methods like LIME, SHAP, and Break Down have become the industry standard for explaining individual predictions. However, these methods share a fundamental underlying assumption: additivity. They assume that a model's prediction can be decomposed into a simple sum of independent feature contributions.

This paper argues that this assumption is often violated in modern machine learning. Complex models like Gradient Boosting Machines (GBMs) and Random Forests derive their power from capturing non-linear interactions between features. When these interactions are forced into an additive framework, the resulting explanations can be inconsistent, unfaithful to the model's true logic, and ultimately misleading to the end-user.

The Problem: Path Dependency and Inconsistency

The core issue with additive explanations is their sensitivity to the order in which features are considered (path dependency). In models with high interaction effects, the contribution assigned to a feature like "Age" might change drastically depending on whether "Passenger Class" was accounted for first.

Figure 3: Break Down explanations for the same instance using different feature orderings, leading to contradictory results.

As shown in Figure 3, changing the sequence of features in a Break Down analysis for a Titanic survival model leads to wildly different conclusions. In Scenario 1, Age appears to have a massive positive contribution (+0.812), while in Scenario 2, that contribution drops significantly (+0.266) because the class effect was processed differently. This inconsistency is not unique to Break Down; the authors prove mathematically that the popular Shapley value (SHAP) is simply an average of all such possible permutations, effectively "smearing" the interaction effects across individual features and hiding the model's true instability.

iBreakDown: Identifying and Visualizing Interactions

To address these shortcomings, the authors introduce iBreakDown. This method moves beyond simple additivity by explicitly identifying and calculating the contributions of feature pairs. The algorithm identifies significant interactions by comparing the joint effect of two features against the sum of their individual marginal effects.

Figure 5: iBreakDown visualization capturing the interaction between Age and Pclass, providing a more faithful explanation.

By incorporating interaction terms (e.g., Age:Pclass) into the visualization, iBreakDown provides a more faithful representation of the model. As seen in Figure 5, the interaction term accounts for a significant portion of the prediction (+0.535), which would have been incorrectly attributed to single features in a standard additive plot. Additionally, the authors propose a bootstrapping approach to visualize uncertainty, adding error bars to explanation plots to signal when a feature's contribution is highly dependent on other variables.

Large-Scale Benchmark and Results

The researchers conducted a large-scale benchmark using the OpenML100 suite, covering 28 binary classification datasets. They trained models with varying interaction depths to test the fidelity of XAI tools.

The findings were striking: 71% of the explanations generated by iBreakDown contained significant interactions. This suggests that in the majority of real-world cases, standard additive tools like SHAP or LIME provide an incomplete or misleading picture of model behavior. While iBreakDown plots are slightly more complex, they significantly reduce explanation uncertainty and offer a higher degree of fidelity to the underlying black-box model.

Conclusion

Additive explanations offer a seductive simplicity, but they often fail to capture the interactive logic that makes complex models effective. Practitioners should be wary of "clean" additive scores and instead utilize tools like iBreakDown that explicitly account for feature interactions and provide measures of explanation uncertainty. The iBreakDown package is currently available for both R and Python to help researchers audit their own models for these hidden inconsistencies.

Summary updated

Abstract

Explainable Artificial Intelligence (XAI)has received a great deal of attention recently. Explainability is being presented as a remedy for the distrust of complex and opaque models. Model agnostic methods such as LIME, SHAP, or Break Down promise instance-level interpretability for any complex machine learning model. But how faithful are these additive explanations? Can we rely on additive explanations for non-additive models? In this paper, we (1) examine the behavior of the most popular instance-level explanations under the presence of interactions, (2) introduce a new method that detects interactions for instance-level explanations, (3) perform a large scale benchmark to see how frequently additive explanations may be misleading.
BibTeX
@misc{gosiewska2019do,
  author = {Alicja Gosiewska and Przemysław Biecek},
  title = {{Do Not Trust Additive Explanations}},
  year = {2019},
  eprint = {1903.11420},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.11420},
}
Figure