Causal Explanations Beyond Feature Saliency

62 concepts56 papers1 proposal

Imagine a deep learning model trained to distinguish between wolves and huskies. If the model mistakenly classifies a husky as a wolf because there is snow in the background, a traditional post-hoc explainability tool—like a saliency map—will faithfully highlight the snow.

This heatmap tells us where the model looked, but it leaves us entirely in the dark regarding the underlying mechanism. Does the model believe snow causes an animal to be a wolf? What would happen if we digitally removed the snow, or transposed the husky into a desert?

This is the core of the causal interpretability gap. Traditional explainability methods rely on statistical correlations between input features and model outputs. They capture observational associations but fail to provide structural, causal explanations. To build truly trustworthy AI, the field is shifting away from static feature attribution toward frameworks that can reason about interventions and counterfactuals.

The Illusion of Feature Importance

For years, feature attribution methods (such as saliency maps and integrated gradients) have served as the standard window into the "black box" of neural networks. However, these methods only highlight correlations within a fixed data distribution.

As Alvin Wan et al. demonstrate in their work on NBDT: Neural-Backed Decision Tree, visual explanations identify influential input regions but fail to reveal the sequential reasoning or logical steps the model took to arrive at a decision. A highlighted patch of pixels does not explain how a model weighs preconditions or handles logical constraints.

In sequential decision-making tasks, such as reinforcement learning, this limitation becomes even more glaring. Sarath Sreedharan et al. show in Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations that visual highlights cannot communicate action preconditions or long-term cost accumulations. Knowing which part of the state space was attended to does not tell a user why an alternative action would have failed.

As Freddy Lecue argues in On The Role of Knowledge Graphs in Explainable AI, post-hoc correlations fail to align with formal human reasoning because humans naturally explain the world through cause-and-effect relationships, not joint probability distributions.

Climbing the Ladder of Causation

To understand why feature attribution falls short, we can look to Judea Pearl’s Ladder of Causation, which divides reasoning into three distinct rungs:

  1. Association ($P(y \mid x)$): Observing what a passive system is doing.
  2. Intervention ($P(y \mid do(x))$): Predicting the outcome when we actively manipulate the system.
  3. Counterfactuals ($P(y_x \mid x', y')$): Imagining alternative realities—asking "What if we had acted differently?"

Standard deep learning architectures operate almost exclusively on the first rung. They optimize for observational likelihood. As Tommaso Salvatori notes in Predictive Coding beyond Correlations, models designed around conditional probabilities are fundamentally unequipped to handle interventions.

This creates a major evaluation problem. In A Critical Review of Causal Reasoning Benchmarks for Large Language Models, Linying Yang et al. reveal that many popular benchmarks mistake simple knowledge retrieval for causal reasoning. Because these evaluations remain restricted to observational data, they fail to test whether a model can predict the outcomes of novel interventions or reason about counterfactual scenarios.

Without climbing higher on the ladder, models risk becoming what Abhinav Joshi et al. term "causal parrots" in their paper COLD: Causal reasOning in cLosed Daily activities—systems that can describe a sequence of events fluently but stumble when asked to reason about what would have happened if an action had been omitted.

The Counterfactual Solution

If highlighting features is insufficient, how do we build explanations that capture true causal logic? The answer lies in counterfactual explanations. Instead of asking "Which features mattered most?", counterfactual interpretability asks: "What is the minimal change required to alter the model's decision?"

Observational Saliency: 
[Input Image] ---> (Model) ---> [Prediction: Wolf]
                       |
                       v
               [Saliency Map: Highlights Snow] (Where did it look?)

Counterfactual Explanation:
[Input Image] ---> [Modify Snow to Grass] ---> (Model) ---> [Prediction: Husky]
                       ^
                       |
               [Actionable Knob] (What changes the outcome?)

Instead of generating a static heatmap, Shusen Liu proposes a generative approach in Generative Counterfactual Introspection for Explainable Deep Learning. By utilizing generative adversarial networks, the framework provides "actionable knobs" that alter specific semantic concepts in an image, showing exactly how a decision boundary shifts.

To make this practical for debugging, Valentyn Boreiko et al. introduce a method in Sparse Visual Counterfactual Explanations in Image Space to find the most localized, realistic changes that flip a classifier's output. This allows developers to catch hidden model biases—such as realizing a model relies entirely on an underwater cage watermark to classify a shark—by observing exactly what modification removes the classification.

In neurosymbolic domains, this can be formalized as a logical abduction problem. In A Logic-based Approach to Contrastive Explainability for Neurosymbolic Visual Question Answering, Thomas Eiter et al. demonstrate that by calculating the minimal scene modifications (such as moving an object or changing its color) required to flip a question-answering model's response, they can provide users with a rigorous, human-verifiable path of causal dependencies.

Designing Causally Constrained Architectures

Moving beyond post-hoc explanations requires changing how models learn in the first place. If an architecture is completely unconstrained, it will naturally exploit any statistical shortcut available in the training set.

To prevent this, researchers are embedding structural causal knowledge directly into model architectures. For example, Matthew James Vowels et al. introduce CaTs and DAGs: Integrating Directed Acyclic Graphs with Transformers for Causally Constrained Predictions. By forcing the Transformer's attention mechanism to respect the structural constraints of a Directed Acyclic Graph (DAG), the model is prevented from learning non-causal patterns, making it significantly more robust to distribution shifts.

Standard Attention:     [Feature A] <---> [Feature B] <---> [Prediction] (Free-form correlation)

Causally Constrained:   [Feature A] ----> [Feature B] ----> [Prediction] (Enforced by DAG)

On the probabilistic side, Florian Peter Busch proposes Structural Causal Circuits, an architecture capable of climbing all three rungs of Pearl's ladder. By combining neural representations with probabilistic circuits, these models can compute exact counterfactual queries in linear time, bridging the gap between deep pattern matching and tractably verifiable causal inference.

As Bernhard Schölkopf argues in Causality for Machine Learning, the hardest open problems in AI—such as out-of-distribution generalization and robust adaptability—are fundamentally problems of causality. Moving from observational feature-mapping to structural, counterfactual explanations is not just a requirement for safety; it is the necessary path forward for machines to truly understand the environments they interact with.

Key Foundational Work

How to Discover a Knowledge Representation for Causal Reasoning by Studying an Expert Physician (1983) — contributed 1 concept elements to this direction.
DANTES: An Expert System for Real-Time Network Troubleshooting (1987) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_causal_reasoning_gap_saliency_map_limita_adb5f74c,
  title = {causal reasoning gap (saliency map limitations)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-reasoning-gap-saliency-map-limitations/rdr_adb5f74c0d934425bc29aab2f408e43f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_causal_reasoning_gap_saliency_map_limita_adb5f74c_v1,
  title = {causal reasoning gap (saliency map limitations)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-reasoning-gap-saliency-map-limitations/rdr_adb5f74c0d934425bc29aab2f408e43f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake