Abstract
We study the faithfulness of an explanation system to the underlying prediction model. We show that this can be captured by two properties, consistency and sufficiency, and introduce quantitative measures of the extent to which these hold. Interestingly, these measures depend on the test-time data distribution. For a variety of existing explanation systems, such as anchors, we analytically study these quantities. We also provide estimators and sample complexity bounds for empirically determining the faithfulness of black-box explanation systems. Finally, we experimentally validate the new properties and estimators.
Introduction: The Faithfulness Gap
As machine learning models are increasingly deployed in high-stakes environments like healthcare and finance, the demand for interpretability has led to the widespread use of "local explainers" such as LIME, SHAP, and Anchors. These systems aim to explain individual predictions by highlighting the features most influential to the model's decision. However, a critical gap remains: how do we know if an explanation is actually faithful to the model's internal logic?
Most existing research evaluates explanations based on human interpretability—whether the explanation makes sense to a person. Yet, an explanation can be highly intuitive while being completely unfaithful to the model. For instance, if a model predicts "rainforest" based on humidity levels, but an explainer attributes the decision to the presence of "trees," the explanation is unfaithful. This paper introduces a formal statistical framework to bridge this gap, defining faithfulness through the lenses of consistency and sufficiency.
Defining Faithfulness: Consistency and Sufficiency
The framework proposes two fundamental properties to quantify how well an explanation system $E$ tracks the behavior of a model $f$:
- Consistency: This property requires that if two different inputs receive the same explanation, the model should ideally produce the same prediction. If $E(x) = E(x')$, then $f(x)$ should equal $f(x')$.
- Sufficiency: This property suggests that an explanation should capture the "essence" of a prediction. If an explanation $E(x)$ identifies a specific property as the reason for a prediction, then any other input $x'$ sharing that property should also yield the same prediction, even if $x'$ is assigned a different explanation.
The Role of Data Distribution and Compression
A key insight of this research is that faithfulness is not a static property of an algorithm; it is deeply dependent on the test-time data distribution. An explanation system might be perfectly faithful for a population in one geographic region but fail in another because the "scope" of an explanation depends on the distribution of the data it encounters.
Furthermore, the authors identify a "verifiability" constraint linked to compression. For an explanation to be empirically verifiable, it must achieve some level of simplification. If an explainer provides a unique, high-precision explanation for every single data point (such as raw, high-precision SHAP values), it becomes mathematically impossible to verify faithfulness without an infinite amount of data. In these cases, consistency and sufficiency scores drop to zero not because the explainer is "bad," but because it provides no generalizable rules.
Analytical and Empirical Insights
The framework was validated across several benchmarks, including the Adult census dataset, movie review sentiment analysis (rt-polaritydata), and various UCI repository datasets. Several key findings emerged:
- SHAP vs. LIME: The authors analytically prove that SHAP possesses perfect consistency by design due to its mathematical axioms, whereas LIME does not necessarily share this property.
- The Necessity of Discretization: For continuous explainers (like gradient-based methods), the authors show that we must discretize the output—for example, by looking only at the "top 5 features"—to make faithfulness measurable in practice. In experiments on the
Bank marketingandAdultdatasets, using a "Sign-of-top-5" discretization boosted SHAP's consistency scores from near 0.0 to over 0.90. - Complexity and Sample Size: Using the
Adultdataset, the researchers demonstrated that as the complexity of the explanation space increases (e.g., a decision tree with more leaves), the number of samples required to accurately estimate faithfulness grows significantly. - Anchors: The
Anchorsexplainer, designed to find high-precision rules, naturally demonstrated high sufficiency but lower consistency compared to white-box methods.
Conclusion and Limitations
This framework provides a unified, statistically grounded way to measure the internal coherence of black-box explanation systems. However, several challenges remain. The proposed estimators require significant amounts of unlabeled data to reach high confidence levels. Additionally, there is a persistent "intelligibility gap": a perfectly faithful explanation might still be too complex for a human to understand. Finally, the choice of how to discretize continuous explanations remains subjective and can significantly influence the resulting faithfulness scores.
Future work should focus on how explanation systems can automatically adapt as data distributions shift, ensuring that faithfulness is maintained over time.
Summary updated