The Faithfulness Interpretability Trade-off in Neural Network Explanations

This direction is split into other directions: interpretability-complexity tradeoff. Its last version stays readable and citable here.

Version 1 of 1238 papers324 conceptsfrozen 2026-10-04

As deep learning models move from research labs into high-stakes environments like healthcare and radiology, the demand for "Explainable AI" (XAI) has intensified. However, researchers have identified a fundamental tension at the heart of the field: the more an explanation makes sense to a human, the less it may actually reflect the model’s internal logic.

This conflict is often framed as a trade-off between faithfulness (how accurately an explanation represents the model's decision-making process) and plausibility (how convincing or understandable the explanation is to a human observer).

The Two Masters of Explainability

In their comprehensive survey, Andreas Madsen, Siva Reddy, and Sarath Chandar distinguish between "human-grounded" and "functionally-grounded" interpretability.

  • Human-groundedness ensures that an explanation is simple, coherent, and fits a user's mental model.
  • Functional-groundedness (faithfulness) ensures that if you change the features the explanation claims are important, the model’s prediction actually changes.

The problem, as Di Jin and Peter Szolovits note in the context of healthcare, is that these two goals often pull in opposite directions. A mathematically faithful explanation of a billion-parameter model might be a high-dimensional vector or a massive computation tree—data that is technically "correct" but cognitively impossible for a doctor to process.

The Plausibility Trap

The most intuitive way to explain a model is through Natural Language Explanations (NLEs)—sentences that justify a decision. However, Pepa Atanasova, Isabelle Augenstein, and colleagues found that these explanations are often "hallucinations." In their Faithfulness Tests for Natural Language Explanations, they revealed that state-of-the-art models generate unfaithful explanations up to 60% of the time. The models learn to sound like a human, but their "reasoning" is a post-hoc justification that doesn't match the internal weights used to make the prediction.

Similarly, in computer vision, Maximilian Dreyer and Reduan Achtibat point out that clean, simple heatmaps often oversimplify the model's logic. A "noisy" heatmap might actually be more faithful to the complex, non-linear interactions the model is using, but humans tend to dismiss noisy visualizations as "bad" explanations.

The Structural Conflict

Why is the truth so messy? One reason is how models process information compared to humans. Humans prefer contiguous structures—we read words in order and look at connected shapes.

Yiming Ju and Kang Liu explored this by removing the "connecting rules" in NLP explanations. Traditional methods force explanations to highlight continuous spans of text. When Ju and Liu allowed for non-contiguous clusters, the faithfulness of the explanations increased significantly. The model was often looking at distant word pairs to make a decision, but forcing the explanation to be a "readable" continuous phrase hid that reality.

This "faithfulness tax" is even more severe in Graph Neural Networks (GNNs). Antonio Longa and Stefano Teso proved a theoretical "trap": for highly expressive GNNs, a perfectly faithful explanation must encompass the entire computational graph. This makes the explanation as large and complex as the model itself, rendering it useless for human interpretation.

If we cannot have perfect faithfulness and perfect simplicity simultaneously, how should we build these systems?

One approach is to treat interpretability as a multi-objective optimization problem. Hazem Torfah and Sanjit A. Seshia propose generating a Pareto-optimal set of interpretations. Instead of one explanation, the system provides a range of options, allowing a user to choose the right balance of fidelity and simplicity for their specific task.

Other researchers are working to ground "plausible" outputs in "faithful" mechanisms. For instance:

  • G-Tex: Shuzhou Yuan and Michael Färber developed a framework that uses GNNs to bake internal reasoning directly into natural language output, bridging the gap between raw highlights and coherent narratives.
  • FFMs: Andreas Madsen and colleagues introduced Inherently Faithfulness Measurable Models, which are designed so that the cost of verifying an explanation's accuracy is mathematically trivial.

Toward a New Paradigm

The field is shifting away from simply making models "look" interpretable. As Mahdi Dhaini found in a survey of industry practitioners, there is deep dissatisfaction with current methods that sacrifice technical accuracy for user-friendliness.

The future of XAI likely lies in what Andreas Madsen calls a "new paradigm" where faithfulness is not a secondary hope, but a primary optimization objective. By acknowledging the trade-off, researchers can build tools that don't just tell us what we want to hear, but accurately reflect the alien logic of the machines we've created.

Papers in this version

Showing 60 of 238 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_faithfulness_interpretability_trade__8a1126e6,
  title = {The Faithfulness Interpretability Trade-off in Neural Network Explanations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-faithfulness-interpretability-trade-off-in-neural-network-explanations/rdr_8a1126e66a3b404eb183fd23a5452479},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_faithfulness_interpretability_trade__8a1126e6_v1,
  title = {The Faithfulness Interpretability Trade-off in Neural Network Explanations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-faithfulness-interpretability-trade-off-in-neural-network-explanations/rdr_8a1126e66a3b404eb183fd23a5452479/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}