Why Post-Hoc Explanations Fail to Reflect Model Logic

261 concepts229 papers3 proposals

Imagine a deep learning model that classifies an MRI scan as showing Alzheimer's disease. To trust this high-stakes decision, a clinician uses a post-hoc explanation tool—like a saliency heatmap—to see which brain regions drove the prediction. The heatmap highlights a specific, clinically relevant area. The clinician nods, satisfied that the model is reasoning correctly.

But did the model actually prioritize that region, or did the explanation tool simply generate a plausible story?

As De Santi et al. and Bennetot et al. point out, popular post-hoc explanation techniques frequently suffer from a fundamental lack of faithfulness (or fidelity). Because these methods generate justifications after a prediction has been made, they often fail to reflect the true internal mechanics of the model. Instead of acting as a transparent window into a black box, post-hoc explainers frequently function like a defense attorney—inventing a logical-sounding justification for a conclusion that may have been reached through entirely different, flawed logic.

Explanatory Inversion: Rationalizing Backwards

To understand why post-hoc methods diverge from reality, we must look at how they process information. The core issue is a phenomenon termed explanatory inversion.

Instead of tracing the forward path of a model's decision-making process, post-hoc methods typically work backward from the final output to find inputs that correlate with it. Tan et al. demonstrate that methods like SHAP and LIME frequently rationalize the output rather than reflecting the true input-output relationship.

Forward Pass (Actual Model Logic):
Input Data ───> [ Hidden Layers / Latent States ] ───> Final Prediction

Explanatory Inversion (Post-Hoc Rationalization):
Plausible Explanation <─── Working Backward ─── Final Prediction

This backward reasoning creates feature attributions that align with human expectations but are causally disconnected from the model's actual computations. The result, as described by Stiglic et al., is a collection of "plausible but wrong" insights. The explanation masks the model's actual reliance on biased, irrelevant, or spurious features, giving users a false sense of security (Glanois et al.).

The Decoupling Problem

This misalignment stems from a structural flaw: post-hoc explainers are entirely decoupled from the model's training phase. As Liu et al. and R. T. et al. argue, because these methods operate as external observers after training is complete, they lack precise knowledge of the model's internal decision boundaries.

Consider a local perturbation method that pokes a black box with slightly altered inputs to map out a local decision boundary. Because it only observes inputs and outputs, it treats the model's internal mechanics as a black box itself.

Han analyzed popular post-hoc methods through the lens of Local Function Approximation (LFA) and proved a troubling theoretical limitation: certain post-hoc techniques fail to recover the true parameters of even simple, completely transparent models when those models are treated as black boxes. If an explainer cannot faithfully extract the logic of a simple linear model, its approximations of deep neural networks are highly suspect.

Latent Misalignment and Spurious Explanations

Another reason post-hoc methods fail is that they typically optimize only for matching the final output, ignoring the internal state of the network.

Zhang et al. show that this focus on output-matching allows explainers to latch onto spurious correlations. For instance, an explanation subgraph in a Graph Neural Network might yield the correct prediction when fed into the model, but its internal latent representations (embeddings) may look completely different from those of the original input graph. Without internal alignment, the explanation is a mathematical mirage—a different path that happens to arrive at the same destination.

Furthermore, post-hoc methods struggle to bridge the gap between low-level features and high-level human understanding. Heritier et al. note that aligning pixel-level importance with human concepts is notoriously unreliable. As Poeta et al. emphasize, post-hoc concept explainers cannot guarantee that the underlying model actually learned or utilized those concepts during training.

The Danger of Deceptive Transparency

When an explanation is unfaithful but highly fluent, it creates a deceptive illusion of transparency. Lima et al. warn that this can lead to "ethics washing," where superficial transparency features satisfy regulatory checklists while obscuring underlying biases or system failures. In high-stakes environments like medicine, an unfaithful explanation can mask critical errors, giving clinicians a false sense of security (Kuijf et al.).

To resolve this crisis of trust, a growing consensus suggests moving away from post-hoc approximations entirely. Instead, researchers advocate for inherently interpretable-by-design architectures—such as prototype learning (Li et al.) or the joint optimization of reasoning and prediction (Wang et al.)—ensuring that the explanation and the decision-making process are one and the same.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_post_hoc_explanation_unfaithfulness_11dc24f2,
  title = {post-hoc explanation unfaithfulness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/post-hoc-explanation-unfaithfulness/rdr_11dc24f2ad784456b9c0173461a154ce},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_post_hoc_explanation_unfaithfulness_11dc24f2_v1,
  title = {post-hoc explanation unfaithfulness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/post-hoc-explanation-unfaithfulness/rdr_11dc24f2ad784456b9c0173461a154ce/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake