The Explanation Game: Explaining Machine Learning Models Using Shapley Values

Luke Merrick, Ankur TalyCD-MAKE 2020 · DOI

TL;DR

This paper argues that existing Shapley-value-based machine learning explanation methods can produce counter-intuitive results due to their underlying "explanation game" formulations. The authors propose a unified framework that clarifies these formulations, enables confidence intervals for attributions, and frames explanations as contrastive comparisons against reference data distributions.

What Problem Does This Paper Address?

Explaining the predictions of complex machine learning models is crucial for trust, debugging, and regulatory compliance. Feature attribution methods, particularly those based on Shapley values from cooperative game theory, have become popular. However, the authors observe that these methods can assign substantial importance to irrelevant features or fail to provide symmetrical attributions when expected. They attribute these issues to the implicit "game" each method defines to represent feature contributions, arguing that the design choices in these games are often poorly justified and lead to misleading explanations. The paper aims to provide a more principled and interpretable approach to Shapley-value-based explanations.

Methods

The core of the paper is the proposal of a unified game formulation for Shapley-value-based explanations. This formulation, $V_{x, D_{ref}}(S)$, is parameterized by a reference distribution $D_{ref}$, allowing existing methods like SHAP (using input distribution $D_{inp}$) and KernelSHAP (using marginal distribution $D_{J.M.}$) to be seen as special cases.

A key conceptual contribution is the decomposition into single-reference games. A single-reference game, $V_{x,r}(S)$, simulates feature absence by replacing feature values with those from a specific reference input $r$. The authors show that the attributions from the unified game are expected Shapley values from these single-reference games. This decomposition offers:

  • Conceptual clarity: Explanations are framed as contrastive comparisons against references.
  • Practical benefits: Enables efficient computation of confidence intervals for attributions, quantifying uncertainty from sampling.

They also introduce the Formulate, Approximate, Explain (FAE) framework:

  • Formulate: Define a contrastive question, specifying relevant references and their distribution $D_{ref}$.
  • Approximate: Sample references from $D_{ref}$ and approximate Shapley values for single-reference games.
  • Explain: Summarize the distribution of approximated Shapley values, potentially using clustering to reveal distinct attribution patterns.

Datasets

The paper uses the following datasets for case studies:

  • UCI Bike Sharing dataset: Used to model hourly bike rentals.
  • UCI Adult Income dataset: Used to predict whether an adult earns more than $50,000 annually.
  • Lending Club dataset: Used to predict loan default.

The models are trained on these datasets, and the explanations are generated for specific predictions within these datasets.

Key Insights and Ideas

  • The "game" matters: The specific way a cooperative game is formulated to represent feature contributions is the root cause of counter-intuitive attributions in existing Shapley-value methods.
  • Contrastive explanations are natural: Framing explanations as comparisons against relevant reference points aligns with cognitive psychology and provides more meaningful insights than absolute attributions.
  • Uncertainty is inherent and quantifiable: Approximating Shapley values and sampling references introduces uncertainty, which can be effectively quantified using confidence intervals derived from the proposed unified framework.
  • FAE framework provides structure: The Formulate, Approximate, Explain framework offers a principled way to generate and interpret contrastive explanations, moving beyond just a "black box" attribution.
  • Mean attributions can be misleading: Averaging attributions across many references can mask important patterns, especially when attributions have opposite signs relative to different references.

Key Results

  • Demonstrated shortcomings: The paper illustrates how existing methods (SHAP, KernelSHAP, QII, IME) can produce unintuitive attributions on simple toy models, such as assigning importance to irrelevant features.
  • Unified framework: The proposed unified game formulation $V_{x, D_{ref}}$ successfully encompasses several existing methods, providing a clear way to understand their differences based on the chosen reference distribution.
  • Confidence intervals: The framework enables the computation of confidence intervals for attributions, quantifying the uncertainty arising from sampling. For example, in the Bike Sharing case study, confidence intervals for "hr" and "workingday" were shown to be wide with limited samples, cautioning against over-reliance on the mean.
  • FAE framework application: Case studies on real-world datasets (Bike Sharing, Adult Income, Lending Club) using the FAE framework revealed more faithful explanations. For instance, by formulating contrastive questions (e.g., "Why was this loan rejected relative to accepted loans?"), the FAE framework highlighted features that were previously under-attributed by existing methods.
  • Clustering for summarization: Applying k-means clustering to single-reference game attributions helped uncover distinct attribution patterns that were "washed out" by averaging in existing methods.

Limitations and Future Work

  • Computational cost: While the paper proposes methods for confidence intervals and a unified framework, computing Shapley values remains computationally expensive, especially for games with many players (features). The paper mentions approximations and sampling, but exact computation is often infeasible.
  • Choice of references: The FAE framework emphasizes the importance of choosing relevant references, but the paper acknowledges that this choice can be subjective and may vary between explainers and explainees.
  • Summarization methods: The paper explores clustering for summarization but suggests that further research is needed on "faithful summarization methods" that accurately represent the attribution distribution.
  • Uncertainty in approximations: The paper notes that while the weighted least squares approximation for Shapley values is used, its unbiasedness is an unproved assumption, which could impact the confidence interval calculations.
  • vcond formulation: The paper notes that the "conditional distribution" game formulation (vcond) cannot be expressed within their unified framework, and discusses the reason for this limitation.

Summary updated

BibTeX
@inproceedings{merrick2020explanation,
  author = {Luke Merrick and Ankur Taly},
  title = {{The Explanation Game: Explaining Machine Learning Models Using Shapley Values}},
  booktitle = {CD-MAKE},
  year = {2020},
  doi = {10.1007/978-3-030-57321-8\_2},
}
Figure