Implications of Model Indeterminacy for Explanations of Automated Decisions

Marc-Etienne Brunet, Ashton Anderson, Richard ZemelNeurIPS 2022 · OpenReview

TL;DR

Even when two machine learning models perform identically on a test set, they can provide wildly different—and even contradictory—explanations for the same individual decision. This paper introduces the concept of "explanatory multiplicity" and demonstrates that standard measures of model uncertainty are poor indicators of whether an explanation is actually robust or just a byproduct of arbitrary model choices.

What Problem Does This Paper Address?

As automated decision-making systems are increasingly deployed in high-stakes areas like credit scoring and insurance, the demand for "explainable AI" (XAI) has grown. We often use local explanation methods, such as Shapley values, to justify why a specific individual was denied a loan.

However, most XAI research assumes we are explaining a single, fixed model. In reality, there is often a "multiplicity of models" (the Rashomon effect) or "underspecification" where many different models achieve nearly identical aggregate performance. The authors ask a critical question: If we have a set of "approximately equivalent" models, do they all give the same explanation for a specific person? If they don't, the "explanation" provided to a user might be more a reflection of a random seed or a specific architecture choice than a fundamental truth about the data.

Methods

The authors formalize the notion of explanatory multiplicity and study it through both theoretical analysis and large-scale experimentation.

  • Model Indeterminacy Types: They distinguish between underspecification (models that differ only due to training randomness like seeds) and the Rashomon effect (a broader set of models, potentially from different classes like Logistic Regression vs. Neural Nets, that all perform well).
Sources of model indeterminacy
  • Explanation Framework: They focus on local, feature-based explanations using Shapley values, which are widely considered the gold standard for fair credit reporting.
  • Consistency Metrics: To quantify the disagreement between models, they introduce:
  • Signed Set Disagreement (SSD): How often the sets of "top-k" most important features differ in sign or identity.
  • Contradicting Direction of Contribution (CDC): The probability that a feature is seen as "helping" a user's case in one model but "hurting" it in another.
  • Theoretical Analysis: They provide a proof in a linear ridge regression setting showing that feature correlation directly increases the likelihood of contradictory explanations.

Datasets

The study evaluates three real-world binary risk assessment datasets from Kaggle:

  1. UCI Credit Card: Predicting credit card default.
  2. Give Me Some Credit: Predicting serious delinquency.
  3. Porto Seguro’s Safe Driver Prediction: Predicting insurance claim filings.

They trained over 1,200 model instances per dataset, including Logistic Regression, Multi-Layer Perceptrons (MLPs), and Tabular ResNets.

Key Insights and Ideas

  • Predictive Multiplicity $\neq$ Explanatory Multiplicity: This is the paper's most striking "aha moment." You might assume that if two models agree on a prediction (low epistemic uncertainty), they must agree on the reason. The authors show this is false. Models can agree perfectly on the output but have completely different gradients or internal logic.
Predictive vs. Explanatory Multiplicity
  • The Shape Matters: Explanations are often driven by the "shape" of the function (its derivatives) rather than just its value at a point. Regions with low predictive uncertainty can still have high gradient variance, leading to inconsistent feature importance.
  • Arbitrary Contradictions: In practice, a user could be told their "number of credit lines" was the primary reason for a denial, while an equally accurate model would have told them that same feature actually helped their score.

Key Results

  • High Disagreement: Across all datasets, the probability of the top-3 features disagreeing (SSD) was over 35% in every experimental setting, and over 50% in two-thirds of them.
  • Alarming Contradictions: The probability of a "top-5" feature actually changing sign (CDC)—meaning the explanation flips from positive to negative—was over 10% in most settings and over 20% in half of them.
  • Zero Correlation: The Spearman correlation between predictive uncertainty and explanation consistency was near zero for two out of three datasets. This means we cannot use standard "uncertainty" flags to warn users when an explanation is likely to be unstable.

Limitations and Future Work

  • Scope of Explanations: The study is limited to Shapley values. While the authors argue the principles apply to other methods like LIME or counterfactual recourse, these were not tested experimentally.
  • Linear Theory: The theoretical analysis is currently limited to linear models; extending this to deep neural networks remains a challenge.
  • Human Baseline: The paper suggests comparing model inconsistency to human decision-maker inconsistency, as humans are also known to provide varying justifications for the same decisions.
  • Robustness Research: The authors call for a focused research effort into developing explanation techniques that are robust to the "trivial or arbitrary" model choices that occur in standard ML pipelines.

Summary updated

Abstract

There has been a significant research effort focused on explaining predictive models, for example through post-hoc explainability and recourse methods. Most of the proposed techniques operate upon a single, fixed, predictive model. However, it is well-known that given a dataset and a predictive task, there may be a multiplicity of models that solve the problem (nearly) equally well. In this work, we investigate the implications of this kind of model indeterminacy on the post-hoc explanations of predictive models. We show how it can lead to explanatory multiplicity, and we explore the underlying drivers. We show how predictive multiplicity, and the related concept of epistemic uncertainty, are not reliable indicators of explanatory multiplicity. We further illustrate how a set of models showing very similar aggregate performance on a test dataset may show large variations in their local explanations, i.e., for a specific input. We explore these effects for Shapley value based explanations on three risk assessment datasets. Our results indicate that model indeterminacy may have a substantial impact on explanations in practice, leading to inconsistent and even contradicting explanations.
BibTeX
@inproceedings{brunet2022implications,
  author = {Marc-Etienne Brunet and Ashton Anderson and Richard Zemel},
  title = {{Implications of Model Indeterminacy for Explanations of Automated Decisions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=LzbrVf-l0Xq},
}
Figure