When we design explainable artificial intelligence (XAI) systems, we often treat interpretability as an intrinsic property of a model's architecture. We assume that replacing a deep neural network with a "glass-box" model—such as a decision tree, a generalized additive model, or a set of case-based prototypes—automatically makes the system understandable.
However, a growing body of research reveals a fundamental bottleneck: human cognitive capacity. An explanation is only as interpretable as a human's ability to process it. When models operate in high-dimensional spaces or generate massive rule sets, their explanations scale in complexity until they exceed our working memory limits. This text explores why traditional interpretability fails at scale and how researchers design around the strict cognitive constraints of the human mind.
The Breakdown of "Glass-Box" Models
The core tension in interpretability is that accuracy often requires capturing high-dimensional relationships, while human understanding requires radical simplicity. When we force a model to be transparent but allow it to scale, its practical intelligibility collapses.
Consider Generalized Additive Models (GAMs), which are widely celebrated for their transparency because they decompose predictions into individual feature shapes without interactions. As Lou et al. demonstrate, when the feature count scales into the hundreds or thousands, the cognitive load of reviewing individual shape functions becomes overwhelming. The sheer volume of data points prevents a human auditor from forming a coherent mental model of the system's global behavior. Similarly, David Rügamer notes that even when higher-order non-linear interactions are mathematically transparent, visualizing and intuitively grasping them remains a significant barrier for human users.
This cognitive breakdown manifests across every major interpretable architecture:
- Case-Based Reasoning: Prototype-based models explain predictions by showing similar examples from the training data. However, Li et al. show that as the number of reference cases grows, evaluating weighted distances to a massive set of prototypes becomes too complex to parse and verify.
- Rule Ensembles and Decision Trees: Logical structures are easy to read when they contain a few branches. But when a model requires hundreds of logical statements to maintain high accuracy, the rules become combinatorially explosive, rendering them practically uninterpretable (Sarkar et al.; Kanamori).
- Linear Feature Importance: Methods that require users to mentally aggregate multiple feature weights impose a heavy cognitive load compared to simple, deterministic "if-then" rules (Ribeiro et al.).
The Bounded Rationality of the Human Auditor
To solve this problem, researchers have begun treating human cognitive limits as hard engineering constraints.
Psychological evidence suggests that humans have a strict "perception budget" and struggle to process interactions between more than a small handful of variables at once. Bruno Apolloni leverages this insight by designing models around an entropic criterion that aggressively prunes features until they meet a human-readable threshold (typically fewer than five variables).
Following a similar intuition, Norrenbrock et al. use discrete optimization to restrict image classifiers to exactly five binary features per class. By forcing this mathematical constraint, they ensure that the model's logic remains strictly within the bounds of human evaluation capacity. As Lomuscio et al. argue, because human decision-makers operate on a limited cognitive budget, an explanation must be sparse and focused to offer actionable or verifiable insights.
[High-Dimensional Data] ──> [Standard XAI] ──> [Massive Explanation] ──> Human Cognitive Overload
│
[High-Dimensional Data] ──> [Budgeted XAI] ──> [Compressed Concepts] ──> Intuitive Understanding
High-Dimensional Explanations as "Data Dumps"
When post-hoc explanation tools (like saliency maps or formal logic reasoners) attempt to be perfectly faithful to a complex model, they often end up producing "data dumps" that are just as complex as the original data.
In fields with massive feature spaces like genomics, Hengjie Yu highlights that standard interpretability methods yield outputs that are too high-dimensional for human experts to interpret, failing to bridge the gap between model mechanics and human understanding. Similarly, exhaustive feature importance maps or pixel-level attributions can overwhelm decision-makers in high-stakes environments, inducing decision paralysis rather than building trust (Feng et al.; Bassan et al.).
Even mathematically rigorous, formal logic-based explanations suffer from this issue. As explored in On The Reasons Behind Decisions, a single decision can be supported by a vast number of minimal "sufficient reasons" that easily exceed human cognitive limits. Meel et al. note that this creates a severe gap between mathematical rigor and practical utility; a proof of correctness is useless if a human cannot read it.
Algorithmic Strategies for Complexity Management
To reconcile the gap between complex models and simple minds, recent literature focuses on three primary mechanisms to compress explanation spaces:
1. Sparsification and Minimization
Instead of providing absolute, exhaustive guarantees, algorithms can search for smaller, probabilistic explanations that capture the lion's share of a model's reasoning. Bounia et al. frame this search as a supermodular minimization problem to efficiently find short, digestible explanations. Mateusz Rychlicki provides fixed-parameter algorithms to compute concise probabilistic explanations for decision trees and sets, ensuring they remain brief enough for human comprehension. In the text and vision domain, An Yan utilizes a learning-to-search method to prune thousands of noisy, large-language-model-generated descriptions down to a tiny, highly descriptive set of natural attributes.
2. Abstraction and Hierarchical Grouping
Humans handle complexity by chunking information into high-level concepts. XAI methods can mimic this by bundling raw features or pixels into abstract representations. Bassan et al. group individual features into "bundles" to provide meaningful abstraction over raw inputs.
In computer vision, Toni et al. and Ayoobi et al. introduce architectures that combine individual prototypical image parts into "super-prototypes" organized within an argumentation framework. This allows the model to present structured, dialectical arguments (supporting and attacking features) without overwhelming the user's information processing capacity. Similarly, Porta et al. organize multi-scale visual prototypes into sparse, logical groups to eliminate redundancy.
3. Localized Interpretability
A model can maintain a highly complex global structure as long as its local predictions are simple. Kentaro Kanamori introduces a framework called LIRE (Locally Interpretable Rule Ensemble) that allows a model to contain thousands of rules globally, while ensuring that any single prediction is explained by only a tiny handful of them. This minimizes the immediate cognitive load required to validate individual decisions. Gao et al. reinforce this concept, noting that because humans cannot synthesize thousands of separate local insights to understand a global model, we must rely on groupwise summaries and localized regions of validity.
The Downstream Costs of Overloading Users
Ignoring cognitive constraints does more than make explanations unappealing—it actively degrades human performance.
When explanations are too visually complex, noisy, or dense, humans fail to derive consistent rules from them, causing a total breakdown in the transfer of understanding from the model to the person (Colin et al.). In fact, Marusich et al. conducted controlled experiments showing that forcing users to process complex explanations increases their cognitive load and reaction times without providing any corresponding boost in decision-making accuracy.
Even advanced interaction paradigms suffer from this ceiling. For instance, presenting "dissenting explanations" (showing a second, conflicting model prediction to prevent human overreliance on AI) can be highly effective. Yet, as Reingold et al. point out, there is a strict threshold: if the system introduces too much disagreement or too many conflicting points, the resulting information overload triggers decision fatigue, completely undermining the benefits of the explanation.
Ultimately, interpretability is a human-centric metric. For an explanation to work, it must respect the mathematical structure of the model, the high-dimensional nature of the data, and—most importantly—the hard limits of human working memory.
Key Foundational Work
Intelligible models for classification and regression (2012) — contributed 2 concept elements to this direction.
Learning Algorithm of Neural Networks Based on the Concept of Possibility and Necessity (1991) — contributed 1 concept elements to this direction.