The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets

Oana-Maria Camburu, Eleonora Giunchiglia, Jakob Foerster, Thomas Lukasiewicz, Phil BlunsomarXiv 2020 · OpenReview

Abstract

Explainability in machine learning is often treated as an objective pursuit, yet even for simple models, a single "ground truth" explanation is often elusive. This paper investigates two prominent feature-based explanation methods—Shapley values and Minimal Sufficient Subsets (MSS)—and demonstrates that they target fundamentally different types of information. Through a series of experiments on transparent, rule-based models, we show that neither method can fully capture a model's decision-making process in isolation. Our findings suggest that the choice of an explainer is a philosophical commitment to a specific definition of feature importance.

Introduction

As neural networks become increasingly integrated into critical decision-making processes, the need for interpretability has led to the development of various feature-based explanation methods. These tools aim to identify which input features, such as specific words in a sentence, are most influential to a model's prediction. However, the field often assumes an objective set of "important" features exists for any given prediction. This research challenges that assumption, revealing that "importance" is subjective and highly dependent on the mathematical framework employed.

Methodology: Comparing Additivity and Necessity

The study compares two dominant but philosophically distinct approaches using rule-based models where the internal logic is known with absolute certainty:

  1. Feature-Additive Explanations (Shapley Values): Rooted in coalitional game theory, Shapley values assign weights to features based on their average marginal contribution across all possible feature combinations. This method prioritizes "fairness" in distributing the total prediction score among the inputs.
  2. Minimal Sufficient Subsets (MSS): This approach identifies the smallest subset of features that, if retained while others are removed, preserves the model's prediction. It focuses on the "core" information required for a specific outcome.

By applying these methods to synthetic sentiment analysis scenarios, the authors pinpoint specific failure modes where each method provides an incomplete or potentially misleading narrative.

Comparative Analysis of Explanation Frameworks

The divergence between Shapley values and MSS reveals several critical insights into how we interpret model behavior, particularly regarding redundancy and context.

Comparison of Shapley and MSS explanations across different rule-based sentiment models. The examples illustrate how Shapley values can assign weight to irrelevant features (averaging trap) and how MSS identifies redundant but sufficient subsets.

The Redundancy Problem

Shapley values struggle with redundant information. If a model requires only one of two positive words (e.g., "good" or "refreshing") to reach a certain sentiment score, Shapley values will split the credit between them. In contrast, MSS can identify that either word alone is sufficient, highlighting the model's internal redundancy and flexibility.

The Averaging Trap

Because Shapley values are calculated as an average over all possible subsets, they can assign significant weight to features that are irrelevant to a specific instance. For example, in a model predicting "smell" sentiment ($m^S$), the word "Tastes" might receive a high Shapley value simply because it is a strong indicator in other contexts, even if it has no impact on the current smell prediction. MSS correctly filters these out by focusing on the minimal set required for the specific output.

Genuine Cancellations vs. Artifacts

In multi-aspect models, positive and negative features may cancel each other out. MSS might ignore both features because the final score remains unchanged if the pair is removed. Shapley values may flag them as important, but they fail to communicate the cancellation effect, potentially confusing the user about the feature's actual role in that specific prediction.

Conclusion and Future Directions

The "taxonomy of struggles" presented in this paper suggests that Shapley values provide a "neighborhood" view—showing average importance—while MSS provides a "pointwise" view—showing what is crucial for a specific instance. Neither is a complete representation of the model's logic.

Future work should move away from seeking a single "best" explainer and instead focus on multi-dimensional explanation frameworks. Combining the additive fairness of Shapley values with the necessity-based insights of MSS could provide a more robust and nuanced understanding of complex black-box models.

Summary updated

Abstract

For neural models to garner widespread public trust and ensure fairness, we must have human-intelligible explanations for their predictions. Recently, an increasing number of works focus on explaining the predictions of neural models in terms of the relevance of the input features. In this work, we show that feature-based explanations pose problems even for explaining trivial models. We show that, in certain cases, there exist at least two ground-truth feature-based explanations, and that, sometimes, neither of them is enough to provide a complete view of the decision-making process of the model. Moreover, we show that two popular classes of explainers, Shapley explainers and minimal sufficient subsets explainers, target fundamentally different types of ground-truth explanations, despite the apparently implicit assumption that explainers should look for one specific feature-based explanation. These findings bring an additional dimension to consider in both developing and choosing explainers.
BibTeX
@misc{camburu2020struggles,
  author = {Oana-Maria Camburu and Eleonora Giunchiglia and Jakob Foerster and Thomas Lukasiewicz and Phil Blunsom},
  title = {{The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets}},
  year = {2020},
  eprint = {2009.11023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.11023},
}
Figure