Navigating Disagreement Among Criteria for Evaluating Explanation Faithfulness

Grant Shanklin, Jiří Němeček, Asia BiegaarXiv 2026 · arXiv

TL;DR

Faithfulness metrics—designed to measure how accurately an explanation reflects a model's behavior—frequently disagree, ranking the same explanations in opposite orders. This paper introduces a "faithfulness functional" to decompose these metrics into three core components, arguing that faithfulness is an "incompletely specified" requirement that must be justified through the specific explanatory, institutional, and normative context of its use.

What Problem Does This Paper Address?

In the world of eXplainable AI (XAI), "faithfulness" is the gold standard. It asks: Does this explanation actually describe how the model works, or is it just a plausible-sounding story?

The problem is that the XAI community has developed dozens of different ways to measure faithfulness (e.g., Infidelity, Sufficiency, Comprehensiveness). Researchers have discovered a sobering reality: these metrics often disagree. An explanation might be "highly faithful" according to one metric but "unfaithful" according to another. This "disagreement problem" leaves developers and auditors in a bind—which metric should they trust?

This paper moves beyond just documenting the disagreement. It seeks to understand why it happens and provides a framework for choosing the right metric for a given situation.

Methods

The authors' primary contribution is a faithfulness functional. They argue that almost any faithfulness metric for local feature-attribution explanations can be broken down into three mathematical components:

  1. Comparison Distribution ($\nu_{x,e}$): This defines which inputs the model is tested against. For example, do we perturb the features the explanation says are important, or the ones it says are unimportant?
  2. Assessment Rule ($\ell_{x,e}$): This defines what a "favorable" model response looks like. If we remove an "important" feature, should the model's output score drop significantly (signed change) or just change at all (absolute change)?
  3. Aggregation Rule ($\mathcal{A}$): This defines how to combine many individual tests into a single score (e.g., taking the mean or calculating a Pearson correlation).

By using this functional, the authors reconstructed eight prominent faithfulness metrics from the literature (summarized in Table 1 of the paper), showing that their differences lie entirely in how they specify these three components.

Datasets

This paper is primarily theoretical and conceptual. It does not introduce new training datasets. However, it uses:

  • A Synthetic Example: A small, 4-feature binary classification setup used to mathematically prove "rank non-equivalence" (that different metrics can reverse the ranking of explanations).
  • Industry Case Study: An analysis of a real-world procedure for generating Adverse Action Notices (AANs) in credit lending, proposed by researchers at Wells Fargo.

Key Insights and Ideas

  • Faithfulness is Incompletely Specified: The authors argue that the general requirement of "faithfulness" is too vague to be a single mathematical formula. Instead, it's a shared concept that requires specific choices to be made before it can be measured.
  • Three Sources of Disagreement: The paper distinguishes between three reasons why metrics might differ, as illustrated in Figure 1:
  • (a) Theoretical Contestation: Disagreeing on what faithfulness even means.
  • (b) Incomplete Specification: Agreeing on the core concept but making different choices about how to apply it in a specific setting. The authors argue this is the primary driver of disagreement in XAI.
    • (c) Indicator Variation: Agreeing on the specific measurement goal but having different (perhaps noisy) ways of calculating it.
Three interpretations of multiple faithfulness operationalizations. The authors defend (b), where plurality stems from different specifications of a shared but incomplete requirement.
  • Contextual Justification: Since faithfulness is incomplete, choosing a metric isn't a neutral technical decision. It requires justification based on:
  • Explanatory considerations: The form of the explanation (e.g., a top-k list of reasons).
  • Institutional considerations: Legal requirements (like the Equal Credit Opportunity Act for AANs).
  • Normative considerations: The ultimate purpose (e.g., helping a consumer improve their credit score).

Key Results

  • Proof of Rank Non-Equivalence: The authors formally prove that common metrics like Infidelity, Sufficiency, and Comprehensiveness are pairwise rank-non-equivalent.
  • Metric Reconstruction: The paper successfully maps eight major faithfulness metrics into the new functional framework, providing a "Rosetta Stone" for comparing XAI evaluations.
  • AAN Case Study: By reconstructing the Wells Fargo B-Shap procedure, the authors show that while some choices (like the reference point for comparison) are well-justified by the legal and educational goals of credit denial notices, other choices (like how to handle feature interactions) remain open and require further normative argument.

Limitations and Future Work

The authors acknowledge that their functional currently focuses on local feature-attribution and rationale-based explanations. Extending this framework to other explanation forms, such as counterfactuals or global explanations, is a clear path for future research.

Furthermore, while the paper provides a framework for how to justify choices, it doesn't provide a "one-size-fits-all" answer for which metric is best. The burden of justification remains with the practitioners and auditors to look closely at their specific application context.

Summary updated

Abstract

Faithfulness requires that algorithmic explanations accurately reflect the model's behavior. Yet existing operationalizations of faithfulness can disagree about whether the same explanation is faithful, leaving developers and auditors to choose among conflicting evaluations. In this paper, we investigate disagreement among evaluation criteria and offer a practical path forward for evaluating explanation faithfulness. We first identify how operationalizations of faithfulness differ. We introduce a faithfulness functional that locates their disagreements in what an explanation claims about the model, which comparisons bear on that claim, and how agreement across those comparisons should be aggregated, and we reconstruct existing evaluation criteria within it. We argue these reconstructions support understanding faithfulness as a common though incompletely specified relation between the model and explanation, so the differences reflect choices left open by the general requirement. Justifying an operationalization then comes down to justifying these choices, which, on our view, draws on explanatory, institutional, and normative considerations. We support this argument with a case study of adverse action notices, reconstructing the evaluation implicit in an industry-proposed procedure and identifying which of its choices the setting justifies and which remain open. Our work gives auditors and practitioners a way to navigate disagreeing evaluations of faithfulness. It also explains why the literature should expect multiple operationalizations, while showing how concrete settings can narrow the choice among them. More broadly, our analysis shows how a shared requirement of faithfulness can support contextually justified evaluations despite disagreement among existing metrics.
BibTeX
@misc{shanklin2026navigating,
  author = {Grant Shanklin and Jiří Němeček and Asia Biega},
  title = {{Navigating Disagreement Among Criteria for Evaluating Explanation Faithfulness}},
  year = {2026},
  eprint = {2611.06846},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06846},
}
Figure