The Interpretability Completeness Tradeoff in Machine Learning

124 concepts100 papers1 proposal

Deep neural networks operate in high-dimensional spaces, mapping intricate patterns to make predictions. When we attempt to explain these models to human users, we encounter a fundamental tension: the interpretability-completeness tradeoff. As outlined by Bau et al., a highly complete explanation captures all technical nuances and internal mechanics of a model, but it quickly becomes too complex for human comprehension. Conversely, simpler explanations provide clarity but sacrifice accuracy, risking becoming merely persuasive rather than transparent.

Because explanations are inherently simplifications of complex systems, Jacovi et al. argue that they necessarily lose information and can never be perfectly faithful. A perfectly faithful representation of a model's high-dimensional internal state is simply too complex for human cognition (Zablocki et al.). Therefore, achieving optimal human understanding requires carefully balancing technical accuracy with human cognitive constraints (Colin et al.).

The Failure of Mathematical Proxies

To make explanations human-readable, researchers often use mathematical proxies like sparsity, short logical formulas, or minimal example sets. However, these proxies frequently fail to align with actual human understanding. Lage et al. demonstrated that models optimized for mathematical simplicity (such as tree depth or sparsity) can still be harder for humans to process or simulate than more complex alternatives. Similarly, Jaakkola et al. note that reducing the length of a logical formula does not guarantee true interpretability, as mathematical conciseness does not automatically minimize cognitive load. This same tension appears in machine teaching, where the smallest possible set of examples needed to mathematically distinguish a concept often lacks the explanatory power required for intuitive human understanding (Simon et al.).

Worse yet, aggressive mathematical simplification can destroy the local accuracy of an explanation. Amparore et al. found that reducing the number of features in local linear explanations often causes a failure in "local concordance"—meaning the simplified explanation predicts a completely different outcome than the actual black-box model at that specific point. When we aggressively simplify a network to make it human-readable, we risk discarding subtle but critical decision pathways (Ayoobi et al.) and missing the nuanced logic or edge-case behaviors captured by the original model (Jansen et al.).

Visual and Structural Fidelity vs. Clarity

The conflict between clarity and truth is highly visible in feature attribution and saliency maps. Methods that generate sharp, clean, and visually sparse sketches are highly interpretable to humans, but Adebayo et al. discovered that they often fail basic sanity checks, producing identical explanations regardless of whether the model weights are trained or completely random. Krakowczyk et al. and Klein confirm that visually simple maps often achieve their clarity by ignoring relevant input features, thereby reducing their faithfulness to the model's actual logic. On the other hand, mathematically faithful attributions tend to produce noisy, complex heatmaps that are far less visually intuitive (Wolf).

A parallel issue exists when extracting symbolic rules or circuits from models. If we increase rule coverage to capture a high percentage of an agent's behavior, the underlying logic grows so complex that it undermines the goal of transparency (Veronese et al.). Faithful symbolic rules extracted from differentiable models are often dense with inequality atoms and long logical chains, making them incredibly difficult for humans to parse (Wang et al.).

In mechanistic interpretability, while simpler logical structures are easy to comprehend, complex representations are required to accurately reflect a model's true computational path (Palumbo et al.). Yet, if we allow alignment maps to become overly complex or non-linear, they can falsely identify structured algorithms in models where they do not exist (Sutter et al.).

The Cognitive Burden of Completeness

Why does completeness degrade interpretability? The primary culprit is human cognitive overhead. When an explanation attempts to be comprehensive, it often triggers information overload. For instance, visual explanations for complex neural architectures like Vision Transformers can highlight too many features, rendering them useless for experts who need rapid insights (Baniecki et al.).

Decomposing neural network decisions into fine-grained concepts or structured graphs can provide high diagnostic accuracy for debugging, but it imposes a heavy cognitive load that makes the system feel less clear than a simple heatmap (Achtibat et al.; Wen et al.). Providing too many reasoning paths introduces noise that degrades clarity (Ma et al.), and deep conceptual metaphors can lose the immediacy required for rapid communication (Xu et al.).

This cognitive burden scales poorly:

  • Attempting to map every local causal dependency results in an exponential increase in connections, making causal graphs too complex for human interpretation (Mueller).
  • As the scope of the task expands, the underlying logic becomes too dense to process (Zhao et al.).
  • Tools providing deep structural insights can paradoxically hinder the rapid identification of model issues by demanding too much mental effort (Song et al.).

Ultimately, high expressiveness becomes a liability if a system cannot translate dense relations into simple, human-readable macros (Staab et al.), leaving a persistent trade-off between explanation granularity and cognitive load (Kumar et al.).

Paradoxes in Human Evaluation

Human interaction with explanations reveals counterintuitive patterns. The authors of Evaluating Saliency Explanations in NLP by Crowdsourcing identified a "Flip" phenomenon: providing more context can actually decrease human accuracy if the model's explanation includes misleading features alongside relevant ones. Conversely, overly simplified feature-based explanations can act as a dangerous cognitive shortcut; they reduce the time humans spend on a task but fail to help them detect model misclassifications because they do not prompt active reasoning (Carton et al.).

Most strikingly, there is a major gap between what users say they want and what actually helps them. Luss et al. found that while participants express a strong preference for minimal, ultra-sparse explanations, they are significantly more accurate at distinguishing functional models from flawed ones when they are provided with longer, more informative, and stable explanation sequences.

Frameworks for Navigating the Tradeoff

To manage this tension, researchers have developed frameworks that treat interpretability and completeness as joint optimization problems rather than mutually exclusive choices:

  • Model Reconciliation: Instead of explaining an entire plan from scratch, Chakraborti et al. propose identifying and correcting only the specific mismatches between the AI's model and the human's mental model. This minimizes cognitive load, though minimal explanations risk becoming misleading if the human's mental model changes later.
  • Simulatability: Poché et al. use automated simulatability (evaluating how well an external agent can predict model outputs based solely on its explanations) as a balancing metric to find the sweet spot where an explanation provides enough detail to be faithful without overwhelming the user.
  • Optimal Information Retention: Yue et al. formalize this balance information-theoretically, optimizing explanations to avoid both redundancy (too much noise) and incompleteness (missing key patterns).
  • Explanation-Based Training: Rather than accepting a model's complexity as fixed, Iwata et al. use differentiable regularizers to train models to inherently produce more faithful post-hoc explanations, though a gap remains between technical faithfulness and human utility.
  • Contextual Filtering: Deng et al. show that while adding contextual information (like coreferences) makes reasoning chains more complete and human-understandable, it can lower performance on strict benchmark metrics that penalize any deviation from a minimal set of facts.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_interpretability_complexity_tradeoff_1a277e24,
  title = {interpretability-complexity tradeoff},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interpretability-complexity-tradeoff/rdr_1a277e247ef34385a7d2b2f520bf21ca},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_interpretability_complexity_tradeoff_1a277e24_v1,
  title = {interpretability-complexity tradeoff},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interpretability-complexity-tradeoff/rdr_1a277e247ef34385a7d2b2f520bf21ca/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake