Designing Neural Networks for Intrinsic and Faithful Interpretability

436 concepts349 papers3 proposals

When we try to understand why a deep learning model made a specific prediction, we typically turn to post-hoc explanation methods. These tools—such as feature saliency maps, LIME, or SHAP—look at a pre-trained, opaque "black box" from the outside and attempt to approximate its behavior.

However, this approach introduces a fundamental vulnerability: the explanation is merely an approximation. As Anton Frederik Thielmann et al. point out, relying on external diagnostic tools creates a layer of separation between the model's actual internal logic and the justification presented to the user. If the post-hoc explainer smooths over nuances or misinterprets a complex interaction, the resulting explanation lacks faithfulness—it tells a plausible story, but not necessarily the true story.

To solve this, a growing body of research focuses on intrinsic interpretability (or ante-hoc design). Instead of trying to explain a black box after training, researchers are designing architectures that are transparent by construction (Luca Longo et al.; Guangming Huang et al.). In these models, the explanation is not a secondary output; it is the exact mechanism through which the model computes its prediction.


Routing Decisions Through Interpretable Concepts

One intuitive way to build an intrinsically interpretable model is to force the network's internal representations to pass through a human-understandable bottleneck.

In a standard neural network, raw inputs are mapped to abstract, high-dimensional latent states before reaching the final classification layer. In contrast, concept-based architectures split the prediction task into two distinct, sequential steps:

  1. Map the raw input to a set of human-understandable properties or concepts.
  2. Compute the final decision strictly as a function of those concepts.
Standard Model:  Input ───────────────────────────► High-Dimensional Latent Space ──► Prediction
Concept Model:   Input ──► [Concept Predictor] ──► Human-Readable Concepts ───────► [Logical Layer] ──► Prediction

As reviewed by Eleonora Poeta et al., embedding these concept representations directly into the neural network architecture during training ensures that the model's internal logic remains transparent. Because the final decision layer can only "see" the predicted concepts, the explanation is mathematically guaranteed to reflect the factors driving the output.

This mechanism is highly effective across different domains. For instance, in reinforcement learning, the Embedded Self-Prediction (ESP) framework developed by Zhengxian Lin et al. constrains an agent's decision-making to be a function of human-understandable future properties (such as "expected damage" or "fuel used"). Similarly, Weixin Chen et al. utilize Neural Probabilistic Circuits to decompose complex decisions into explicit attribute probabilities followed by structured logical reasoning.


Sample-Specific Local Linear Generators

Linear models are universally considered interpretable because we can look directly at their weights to see how each feature influences the prediction. However, simple linear models lack the capacity to capture the highly non-linear relationships present in real-world data.

Intrinsic interpretability frameworks bridge this gap by using deep neural networks to generate sample-specific linear models.

Input (x) ───┬───► [Neural Weight Generator] ───► Sparse Weights (w_x)
             │                                           │
             └───────────────────────────────────────────┴───► Linear Predictor (w_x · x) ──► Output

In the Neural Generators of Sparse Local Linear Models (NGSLL) framework proposed by Yuya Yoshikawa et al., a deep network acts as an auxiliary processor that inspects an input and outputs a sparse set of linear weights tailored specifically to that input. The final prediction is then computed by taking the dot product of these generated weights and the input features.

This architecture delivers the best of both worlds:

  • High Accuracy: The weight generator can learn incredibly complex, non-linear functions across the entire dataset.
  • Perfect Faithfulness: For any individual sample, the final prediction is strictly a linear combination of its features. The generated weights are the explanation, completely eliminating the risk of post-hoc distortion.

In-Processing and Self-Explaining Attention

Another prominent strategy involves modifying the internal components of standard architectures—such as attention mechanisms or gradients—so that they naturally produce clean, reliable explanations during a single forward pass.

In standard Transformers, attention weights are often distributed globally, making them noisy and difficult to interpret as direct feature importance maps. To fix this, Jean-Flavien Bussotti et al. introduce architectures that directly modify and refine attention weights during the training phase, integrating the explanation objective into the core optimization task. By enforcing sparsity and structural constraints, the model learns to isolate relevant evidence as part of its primary computation.

This "in-processing" approach can take several forms:

  • Single-Pass Feature Maps: Saifullah Saifullah et al. replace standard pooling layers with specialized internal attention mechanisms, allowing the network to output faithful feature importance maps simultaneously with the prediction, removing the need for slow post-hoc visualization passes.
  • Path Selection: For graph-structured data, Wenda Li et al. developed modules that dynamically select the most critical message-passing paths during the inference cycle, making the network self-explainable.
  • Regularized Gradients: Instead of altering the architecture, Shizhan Gong et al. utilize norm-regularized adversarial training to constrain the model's loss landscape. This mathematical duality forces the model's natural gradients to be inherently sparse, stable, and structured, ensuring that vanilla gradient-based attributions are perfectly aligned with the internal logic.

Eliminating the Performance Trade-Off

A common misconception in machine learning is that we must choose between a highly accurate black box and a simplistic, interpretable model. Intrinsic interpretability research explicitly challenges this dichotomy.

By embedding explanation mechanisms directly into the inference cycle—whether through game-theoretic attribution layers (Hengshu Zhu et al.), sparse feature masking (Jacob Yoke Hong Si et al.), or hierarchical topic modeling (Hanqi Yan et al.)—these architectures demonstrate that transparency can be achieved without sacrificing predictive power. When the explanation is designed as the engine of prediction rather than an afterthought, the model naturally learns to align its internal reasoning with human-understandable structures.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 344 papers
  2. Version 1 · 2026-10-04 · 329 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_intrinsic_interpretability_interpretabil_b50f55b8,
  title = {intrinsic interpretability (interpretability by design)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/intrinsic-interpretability-interpretability-by-design/rdr_b50f55b87668475fa29b816d8954a30b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_intrinsic_interpretability_interpretabil_b50f55b8_v2,
  title = {intrinsic interpretability (interpretability by design)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/intrinsic-interpretability-interpretability-by-design/rdr_b50f55b87668475fa29b816d8954a30b/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake