Towards a Theory of Faithfulness: Faithful Explanations of Differentiable Classifiers over Continuous Data

Nico Potyka, Xiang Yin, Francesca ToniarXiv 2022 · DOI · arXiv

Introduction

Explainable AI (XAI) aims to provide transparency into how complex machine learning models make decisions. A central requirement for any explanation is "faithfulness"—the degree to which an explanation accurately reflects the model's internal logic. Despite its importance, faithfulness has often remained a vague concept, typically measured by how well a simple proxy model (like a linear regressor) approximates a complex "black-box" model in a local neighborhood. This paper introduces mathematical rigor to the concept of faithfulness, specifically for differentiable classifiers operating on continuous data.

Problem Statement: The Flaw in Proxy-Based Faithfulness

Current XAI methods often rely on the assumption that a good local approximation of a model's output implies a faithful explanation. However, the authors argue that a proxy model might achieve high predictive accuracy for the wrong reasons, leading to misleading explanations. This is particularly critical in high-stakes domains like medicine or finance, where understanding the true drivers of a model's prediction is paramount. The paper seeks to replace intuitive notions of faithfulness with formal, testable definitions derived from real analysis.

Theoretical Framework: Defining Faithfulness

The authors propose two formal desiderata for feature attribution methods:

  1. Qualitative Faithfulness: A method is qualitatively faithful if a positive feature score implies that increasing that feature's value locally increases the model's output probability.
  2. Quantitative Faithfulness: The magnitude of the attribution score should be proportional to the actual change in the model's output.

Using tools from real analysis, the authors prove that Gradient-scoring (using partial derivatives) is the theoretical gold standard for differentiable models.

  • Propositions 1 & 3 establish that Gradient-scoring is inherently qualitatively faithful. Any other method satisfying this property must be qualitatively equivalent to the gradient.
  • Propositions 4 & 5 demonstrate that Gradient-scoring is the unique method satisfying local quantitative faithfulness, as it represents the first-order Taylor expansion—the best linear approximation of the model.

Experimental Evaluation

The theory was tested using three continuous datasets from the UCI Machine Learning Repository: Banknote Authentication (BKNT), Breast Cancer Wisconsin Diagnostic (WDBC), and Pima Indians Diabetes (PIMA). The authors compared Gradient-scoring against popular methods like LIME and SHAP using Logistic Regression (where ground truth weights are known) and Multilayer Perceptrons (MLPs).

Key Results

  • Accuracy: In Logistic Regression, Gradient-scoring achieved a perfect 1.000 Spearman rank correlation with true feature weights. LIME followed closely (~0.97), while SHAP struggled significantly, often showing near-zero or even negative correlations (e.g., -0.073 on BKNT).
  • Efficiency: Gradient-scoring is orders of magnitude faster. Gradients for MLPs were computed in ~0.2-0.3 ms, whereas SHAP required over 3,000 ms.

Key Insights: Local vs. Global Faithfulness

A major takeaway is that for non-linear models, global faithfulness is often an impossible goal. A feature's influence can change drastically depending on the input region.

Comparison of monotonic (left) and non-monotonic (right) model behaviors. The right plot illustrates how a feature's influence (slope) can change sign, necessitating local rather than global explanations.

The "noise" often attributed to gradient-based methods in fields like Computer Vision is likely a byproduct of applying them to discrete data (pixels). On truly continuous data, the gradient is the most precise tool. Furthermore, methods like LIME are essentially expensive, noisy approximations of the gradient. If a model is differentiable, calculating the derivative directly is both more accurate and more efficient.

Conclusion and Future Work

This research establishes that for differentiable models on continuous data, the gradient is not just a baseline but the most faithful explanation method. While currently limited to differentiable models and continuous features, the authors suggest that LIME may serve as a natural extrapolation for non-differentiable cases. Future work aims to extend this framework to capture feature interactions through higher-order derivatives (the Hessian matrix).

Summary updated

Abstract

There is broad agreement in the literature that explanation methods should be faithful to the model that they explain, but faithfulness remains a rather vague term. We revisit faithfulness in the context of continuous data and propose two formal definitions of faithfulness for feature attribution methods. Qualitative faithfulness demands that scores reflect the true qualitative effect (positive vs. negative) of the feature on the model and quanitative faithfulness that the magnitude of scores reflect the true quantitative effect. We discuss under which conditions these requirements can be satisfied to which extent (local vs global). As an application of the conceptual idea, we look at differentiable classifiers over continuous data and characterize Gradient-scores as follows: every qualitatively faithful feature attribution method is qualitatively equivalent to Gradient-scores. Furthermore, if an attribution method is quantitatively faithful in the sense that changes of the output of the classifier are proportional to the scores of features, then it is either equivalent to gradient-scoring or it is based on an inferior approximation of the classifier. To illustrate the practical relevance of the theory, we experimentally demonstrate that popular attribution methods can fail to give faithful explanations in the setting where the data is continuous and the classifier differentiable.
BibTeX
@misc{potyka2022towards,
  author = {Nico Potyka and Xiang Yin and Francesca Toni},
  title = {{Towards a Theory of Faithfulness: Faithful Explanations of Differentiable Classifiers over Continuous Data}},
  year = {2022},
  eprint = {2205.09620},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.09620},
}
Figure