In high-stakes domains like medicine, autonomous driving, and finance, a model’s accuracy is only half the story. As Stuart Russell 0001 and Sabine Hauert argue in Robotics: Ethics of artificial intelligence, the "black box" nature of deep learning creates a fundamental barrier to trust. If a clinician cannot see the reasoning behind a recommendation, they cannot safely act on it.
Historically, researchers have faced a "performance-explainability trade-off." To make a model interpretable, you typically had to simplify it—using a shallow decision tree instead of a deep neural network—which inevitably led to a "performance tax" where accuracy dropped as clarity increased. Recent research, however, focuses on intrinsic interpretability: designing architectures that are transparent by design while maintaining the predictive power of unconstrained models.
The Bottleneck Strategy
The most common way to build an interpretable model is to force its internal logic through a human-readable "bottleneck." If the model must use specific, predefined concepts to reach a conclusion, we can inspect that intermediate step to understand its reasoning.
In Intermediate Entity-based Sparse Interpretable Representation Learning, Diego García-Olano and colleagues introduced ItsIRL, which forces a neural network to pass information through a layer of 68,000 fine-grained entity types. This allows the model to remain high-performing while providing a clear map of which entities (e.g., "cardiovascular disease" or "medication side effect") triggered a prediction.
Similarly, Hamid Palangi and Paul Smolensky developed the Tensor Product Recurrent Network (TPRN) in Deep Learning of Grammatically-Interpretable Representations Through Question-Answering. Instead of messy, opaque vectors, the TPRN uses "Tensor Product Representations" to decompose language into discrete symbols (meanings) and roles (grammar). This allows the model to match state-of-the-art performance while its internal states remain mathematically translatable into linguistic concepts like "subject" and "verb."
Reasoning by Prototype
Another approach to closing the performance gap is "case-based" reasoning. Humans often explain decisions by pointing to examples: "This image is a bird because it has a beak similar to this specific woodpecker I've seen before."
Architectures like PxGNN, proposed by Enyan Dai and Suhang Wang in Towards Prototype-Based Self-Explainable Graph Neural Network, bake this logic into the model itself. By learning representative "prototype graphs" for each class, the model makes predictions based on how closely a new input matches these learned exemplars. This provides "faithful" explanations—the reasons given are the actual basis for the prediction—without the accuracy loss typically seen when trying to explain a model after it has already been trained.
Differentiable Logic
A third direction involves making "glass-box" models—like decision trees or additive models—compatible with the powerful optimization tools of deep learning.
- Differentiable Trees: In Interpretable Reinforcement Learning via Differentiable Decision Trees, Andrew Silva and Taylor W. Killian show that decision trees can be trained using gradient descent. This allows them to match the performance of neural networks in complex reinforcement learning tasks while remaining readable as a series of "if-then" statements.
- Additive Models: Yin Lou and Rich Caruana demonstrated in Intelligible models for classification and regression that Generalized Additive Models (GAMs), when powered by modern gradient-boosted trees, can achieve near-state-of-the-art accuracy. Because these models treat each feature independently, they are inherently interpretable: you can simply plot the "shape function" of a feature to see exactly how it affects the outcome.
Why It Works: Moving Beyond Post-Hoc Explanations
The shift toward intrinsic interpretability is a reaction to the failures of "post-hoc" explanation methods. As noted in A Survey on Explainable Reinforcement Learning by Shunyu Liu, applying a separate tool to explain a black box risks generating "hallucinated" justifications that don't actually reflect the model's internal logic.
By contrast, models like Adaptive Transformer Programs (ATP), introduced by Quoc-Vinh Lai-Dang in Adaptive Transformer Programs: Bridging the Gap Between Performance and Interpretability in Transformers, use a smooth optimization transition to turn a high-performance Transformer into concise, interpretable pseudocode.
The intuition across all these papers is the same: interpretability should not be an afterthought or a separate module. By integrating transparency into the architecture's objective function—whether through symbolic bottlenecks, prototypes, or differentiable logic—researchers are proving that we don't have to choose between a model that works and a model we understand.