TL;DR
Even when two machine learning models perform identically on a test set, they can provide wildly different—and even contradictory—explanations for the same individual decision. This paper introduces the concept of "explanatory multiplicity" and demonstrates that standard measures of model uncertainty are poor indicators of whether an explanation is actually robust or just a byproduct of arbitrary model choices.
What Problem Does This Paper Address?
As automated decision-making systems are increasingly deployed in high-stakes areas like credit scoring and insurance, the demand for "explainable AI" (XAI) has grown. We often use local explanation methods, such as Shapley values, to justify why a specific individual was denied a loan.
However, most XAI research assumes we are explaining a single, fixed model. In reality, there is often a "multiplicity of models" (the Rashomon effect) or "underspecification" where many different models achieve nearly identical aggregate performance. The authors ask a critical question: If we have a set of "approximately equivalent" models, do they all give the same explanation for a specific person? If they don't, the "explanation" provided to a user might be more a reflection of a random seed or a specific architecture choice than a fundamental truth about the data.
Methods
The authors formalize the notion of explanatory multiplicity and study it through both theoretical analysis and large-scale experimentation.
- Model Indeterminacy Types: They distinguish between underspecification (models that differ only due to training randomness like seeds) and the Rashomon effect (a broader set of models, potentially from different classes like Logistic Regression vs. Neural Nets, that all perform well).
- Explanation Framework: They focus on local, feature-based explanations using Shapley values, which are widely considered the gold standard for fair credit reporting.
- Consistency Metrics: To quantify the disagreement between models, they introduce:
- Signed Set Disagreement (SSD): How often the sets of "top-k" most important features differ in sign or identity.
- Contradicting Direction of Contribution (CDC): The probability that a feature is seen as "helping" a user's case in one model but "hurting" it in another.
- Theoretical Analysis: They provide a proof in a linear ridge regression setting showing that feature correlation directly increases the likelihood of contradictory explanations.
Datasets
The study evaluates three real-world binary risk assessment datasets from Kaggle:
- UCI Credit Card: Predicting credit card default.
- Give Me Some Credit: Predicting serious delinquency.
- Porto Seguro’s Safe Driver Prediction: Predicting insurance claim filings.
They trained over 1,200 model instances per dataset, including Logistic Regression, Multi-Layer Perceptrons (MLPs), and Tabular ResNets.
Key Insights and Ideas
- Predictive Multiplicity $\neq$ Explanatory Multiplicity: This is the paper's most striking "aha moment." You might assume that if two models agree on a prediction (low epistemic uncertainty), they must agree on the reason. The authors show this is false. Models can agree perfectly on the output but have completely different gradients or internal logic.
- The Shape Matters: Explanations are often driven by the "shape" of the function (its derivatives) rather than just its value at a point. Regions with low predictive uncertainty can still have high gradient variance, leading to inconsistent feature importance.
- Arbitrary Contradictions: In practice, a user could be told their "number of credit lines" was the primary reason for a denial, while an equally accurate model would have told them that same feature actually helped their score.
Key Results
- High Disagreement: Across all datasets, the probability of the top-3 features disagreeing (SSD) was over 35% in every experimental setting, and over 50% in two-thirds of them.
- Alarming Contradictions: The probability of a "top-5" feature actually changing sign (CDC)—meaning the explanation flips from positive to negative—was over 10% in most settings and over 20% in half of them.
- Zero Correlation: The Spearman correlation between predictive uncertainty and explanation consistency was near zero for two out of three datasets. This means we cannot use standard "uncertainty" flags to warn users when an explanation is likely to be unstable.
Limitations and Future Work
- Scope of Explanations: The study is limited to Shapley values. While the authors argue the principles apply to other methods like LIME or counterfactual recourse, these were not tested experimentally.
- Linear Theory: The theoretical analysis is currently limited to linear models; extending this to deep neural networks remains a challenge.
- Human Baseline: The paper suggests comparing model inconsistency to human decision-maker inconsistency, as humans are also known to provide varying justifications for the same decisions.
- Robustness Research: The authors call for a focused research effort into developing explanation techniques that are robust to the "trivial or arbitrary" model choices that occur in standard ML pipelines.
Summary updated