Introduction
As machine learning models grow increasingly complex, the AI community relies heavily on post-hoc explanation systems—such as LIME, SHAP, or Anchors—to interpret model predictions. However, a critical question remains: do these human-readable explanations actually reflect the underlying model's true decision-making process?
This paper introduces a formal, quantitative framework to measure the "faithfulness" of machine learning explanations, providing a rigorous evaluation method for both white-box and black-box systems across varying data distributions.
A Framework for Faithfulness
The authors decompose faithfulness into two measurable, mathematical properties:
- Consistency: If two different inputs yield the exact same explanation, do they also receive the same prediction from the underlying model?
- Sufficiency: If an explanation $\pi$ justifies a prediction for an input $x$, does that explanation guarantee the same prediction for any other input $x'$ where $\pi$ also applies?
For black-box models where internal logic is hidden, the authors develop a statistical estimator ($\hat{M}$) to calculate these scores using unlabeled test samples.
The Verifiability Paradox and Discretization
A core technical insight of the paper is the role of compression and the "verifiability paradox." If an explainer generates a completely unique explanation for every single data point, it becomes statistically unverifiable. You cannot prove an explanation is consistent if you never observe it applied to more than one instance.
To evaluate continuous explainers (like feature importance vectors in SHAP), the authors propose discretization methods to group similar explanations together, effectively compressing the explanation space so that statistical verification becomes possible.
Key Insights and Results
The framework was validated on several benchmarks, including the Adult census dataset, rt-polaritydata, and various UCI datasets. The experiments revealed several critical insights:
- Faithfulness is Data-Dependent: An explanation system's faithfulness is not absolute; it depends heavily on the test-time data distribution. An explainer might be highly faithful for one demographic but fail when applied to another.
- Sample Complexity Increases with Detail: As the "explanation space" grows (for example, a decision tree with more leaves), the number of samples required to accurately estimate faithfulness increases significantly.
- Discretization Recovers Verifiability: For explainers like SHAP, using raw continuous values often results in a consistency score of 0.0 because every explanation is unique. By applying discretization techniques (such as the "1-FP" method, which rounds to one floating-point decimal), researchers can recover meaningful faithfulness scores.
- Trade-offs in Scoped Rules: For rule-based explainers like Anchors, increasing the "precision threshold" improves sufficiency (the explanation is more accurate) but simultaneously increases the number of unique explanations. This makes the system harder to verify, highlighting the tension between explanation precision and verifiability.
- SHAP vs. LIME: The authors analytically prove that SHAP has perfect consistency by design due to its mathematical axioms, whereas LIME does not necessarily share this property.
Limitations and Future Directions
While the framework provides a robust mathematical foundation for faithfulness, it has limitations. The optimal method for discretization (e.g., rounding values versus using signs) remains empirical and varies by dataset. Furthermore, the current estimators struggle with truly continuous explanation spaces.
The authors suggest that future work could incorporate Lipschitz assumptions—the premise that similar inputs should yield similar explanations—to build more sophisticated estimators. Finally, this work focuses strictly on mathematical faithfulness; whether these faithful explanations are actually interpretable or useful to human users remains a separate, vital area of research.
Summary updated