Overcoming the Interpretability Tax in Neural Networks

This direction is merged into another direction: accuracy-interpretability trade-off. Its last version stays readable and citable here.

Version 1 of 144 papers45 conceptsfrozen 2026-10-04

In high-stakes domains like medicine, finance, and autonomous systems, the "black box" nature of deep learning is a liability. While a model might achieve state-of-the-art accuracy, its reasoning is often buried in high-dimensional weight matrices that defy human intuition. This has led to a central tension in the field: the interpretability tax.

As Yu Zhang 0111 and colleagues note in their survey on interpretability, there is a persistent inverse correlation between a model's predictive power and its transparency. When we force a network to be understandable, we often break the very non-linear representations that make it powerful.

The Concept Bottleneck

The most direct way to make a model "interpretable by design" is to force its decision-making process through a human-understandable intermediate layer, known as a Concept Bottleneck Model (CBM). Instead of mapping pixels directly to a "Pathology" label, a CBM first predicts intermediate concepts—such as "breathiness" or "strain" in voice disorder detection, as explored by Davide Ghia and Gabriele Ciravegna in their recent work.

The intuition is simple: if a human can verify the intermediate steps, they can trust the final output. However, this creates a "bottleneck" in the literal sense. Diego García-Olano and Byron C Wallace demonstrate in ItsIRL that restricting a model to a fixed set of entities often incurs a performance penalty. The model loses the ability to use "residual" information—those subtle, non-verbalizable patterns that contribute to accuracy but don't fit into a tidy human category.

Self-Explainable Architectures

To avoid the interpretability tax, researchers are moving away from "post-hoc" explanations (which try to explain a model after it is trained) toward intrinsic interpretability. The goal is to make the forward pass itself the explanation.

One approach is to modify the architecture to produce "evidence maps" alongside predictions. Kerol Djoumessi and Philipp Berens introduced Soft-CAM, which replaces standard classification heads with a "class evidence" layer. By integrating the explanation into the forward pass, they achieve localization precision without the accuracy drop typically associated with transparency.

Similarly, TesNet, developed by Huafeng Liu and Liping Jing, uses Grassmann manifolds to ensure that learned visual concepts are mathematically orthogonal. This prevents "concept bleed," where the model's internal logic becomes muddled, allowing for clear "scoring sheets" for every decision.

Inducing Structural Modularity

Another perspective on the black box problem is that neural networks are too "entangled." In a standard network, every neuron can potentially influence every other neuron, creating a chaotic web.

Anna Soligo and Pietro Ferraro have proposed pipelines for Reinforcement Learning that force agents to organize their internal weights into functional modules. By penalizing long-distance connections, they create "brain-like" structures where specific sub-networks handle specific tasks (like X-axis vs. Y-axis navigation). This modularity allows for "surgical" interventions—if a model is failing, developers can identify and fix the specific module responsible.

Scaling Interpretability with Foundation Models

Historically, the biggest hurdle for CBMs was the need for massive, manually labeled concept datasets. Subhro Das and others have addressed this with Label-free CBMs. By using foundation models like GPT-3 to generate concept libraries and CLIP to align them with images, researchers can now build interpretable models at the scale of ImageNet without a single manual label.

This shift suggests that the interpretability tax is not a fundamental law of AI, but a design challenge. As Vinitra Swamy argues with InterpretCC, by using conditional computation to activate only the most relevant "expert" sub-networks, we can achieve models that are both sparse enough for humans to understand and dense enough to remain highly accurate.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_overcoming_the_interpretability_tax_in_n_6235f109,
  title = {Overcoming the Interpretability Tax in Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-interpretability-tax-in-neural-networks/rdr_6235f109ba7d4a759b669862420a6fc7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_overcoming_the_interpretability_tax_in_n_6235f109_v1,
  title = {Overcoming the Interpretability Tax in Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-interpretability-tax-in-neural-networks/rdr_6235f109ba7d4a759b669862420a6fc7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}