Deep neural networks owe their remarkable performance to non-linear transformations. By stacking layers of matrix multiplications and non-linear activations, a network can warp data spaces to find complex, high-dimensional boundaries. However, this expressive power comes at a steep cost: inside a fully non-linear black box, features interact in tangled ways, making it nearly impossible to isolate exactly how much a single input or concept contributed to the final output.
To resolve this tension, a prominent research direction focuses on enforcing a linear aggregation bottleneck within the model architecture. Instead of allowing non-linearities to run all the way to the final prediction, these architectures decouple the model into two parts: a highly non-linear feature extractor that maps raw inputs to high-level, human-understandable concepts, and a strictly linear or additive layer that aggregates these concepts into a final decision.
Why Linear Aggregation Works
To understand why this design pattern guarantees interpretability, consider the mathematical simplicity of a linear combination:
$$y = \sum_{i} w_i x_i$$
In a standard deep network, the impact of an early feature depends entirely on the states of all other features due to downstream non-linearities. But in a linear bottleneck, the marginal contribution of feature $x_i$ to the output $y$ is governed strictly by its weight $w_i$, completely independent of any other feature values. This independence yields several powerful properties:
- Exact Attribution: The product $w_i x_i$ represents the exact, unclouded evidence provided by feature $i$. There are no hidden feature interactions or downstream distortions.
- Proportionality and Linearity: As shown by Agarwal in a study on explanation mechanics, inherently linear aggregations naturally satisfy the mathematical properties of linearity and proportionality, meaning the explanation perfectly reflects the underlying model weights without approximation errors.
- Direct Auditing: Practitioners can directly inspect, rank, or even manually adjust the weights $w_i$ to debug the model or align it with domain knowledge.
By placing this linear layer at the very end of the network, researchers create a "glass-box" decision step fed by a "black-box" feature extractor.
Decoupling Representation from Decision Logic
A common implementation of this principle is to use powerful deep learning models—such as Large Language Models (LLMs) or Convolutional Neural Networks (CNNs)—solely to extract high-level semantic concepts, which are then combined via a transparent linear head.
For instance, in clinical natural language processing, McInerney et al. developed the CHiLL framework, which uses an LLM to extract zero-shot, human-readable clinical concepts from raw medical notes. These concepts are then passed directly to a simple linear model. Because the final decision relies entirely on linear coefficients, clinicians can verify exactly which extracted facts drove a high-stakes medical prediction.
A similar strategy applies to computer vision. Wickramanayake et al. introduced a concept layer into CNNs that forces the network to learn visual features explicitly mapped to human phrases (e.g., "yellow petals"). By applying global average pooling to this layer followed by a linear classification head, the model's final decision becomes a transparent, weighted sum of individual concept activations.
Even in complex time-series forecasting, the same pattern holds. Jiang et al. designed a short-term load forecasting model that processes multi-scale temporal trends through specialized deep sub-modules, but aggregates their outputs using a final linear combination layer. This creates an interpretable bottleneck where the learned weights serve as global importance scores for different temporal scales and auxiliary features.
Inherent Linearity vs. Post-Hoc Approximations
Many popular interpretability tools are post-hoc, meaning they attempt to approximate the behavior of an already-trained black box by sampling local regions. However, these approximations can be unfaithful or noisy. Linear aggregation bottlenecks avoid this by making the internal weights the explanation itself.
For example, Chu et al. demonstrated that for piecewise linear neural networks (such as those using ReLU activations), an exact, closed-form interpretation can be calculated within the specific convex polytope active for a given input. This avoids local sampling errors entirely by extracting the true, underlying linear function the model is executing at that exact point.
Taking this concept further in computer vision, Böhle et al. introduced B-cos Vision Transformers. Instead of relying on post-hoc attention maps, they redesigned the transformer architecture to function as a dynamic linear transform. During a forward pass, the specific weights applied to the input pixels serve as a high-resolution explanation map of both positive and negative evidence, ensuring the explanation is perfectly faithful to the model's internal logic.
Scaling to Generalized Additive Models
While pure linear models are highly transparent, they can sometimes oversimplify complex relationships where a single feature's impact isn't strictly straight-line. To capture non-linear feature curves without sacrificing the clarity of linear aggregation, researchers frequently turn to Generalized Additive Models (GAMs):
$$y = \sum_{i} f_i(x_i)$$
In a GAM, each feature $x_i$ is processed by its own non-linear function $f_i$, but the individual outputs are still summed together linearly. This preserves the core benefit of linear aggregation: the features do not interact, meaning the contribution of $x_i$ can still be completely isolated and visualized independently.
The authors of a study on mental health classification (Lee et al.) utilized GAMs to map specialized linguistic behaviors to final predictions, ensuring that the contribution of each behavioral feature remains fully auditable by domain experts.
Similarly, Wang et al. leveraged Explainable Boosting Machines (EBMs)—a high-performance tree-based GAM—to build interactive recourse systems. Because the global model structure is additive, the system can easily compute exactly how a user needs to alter their input features to change an algorithmic outcome, providing clear, actionable paths for recourse.
Cross-Modality Universality
The pattern of linear concept aggregation repeats across diverse data modalities, proving that transparency does not require domain-specific mathematical formulations:
- Medical Imaging: Djoumessi et al. developed an inherently interpretable model for early diabetic retinopathy screening by calculating the final diagnosis as a simple average of local image patch scores. This "bag-of-patches" approach maintains a direct mathematical link between local visual evidence and the global output, preventing the model from relying on hidden spatial relationships.
- Natural Language Processing: Sun explored the Backpack language model architecture, which generates token predictions by calculating a weighted sum of static sense vectors. This linear aggregation ensures that the contextualized contribution of each word sense is mathematically separable and transparent.
- Relational and Graph Data: In the domain of universal algebra, Giannini et al. constructed an interpretable graph network (iGNN) by applying a linear model directly to the frequencies of discrete structural concepts identified during message passing. This allowed the model to map global algebraic property predictions directly back to specific local subgraphs, helping mathematicians discover necessary and sufficient structural conditions for mathematical conjectures.
Ultimately, linear aggregation bottlenecks prove that deep learning models do not have to be uninterpretable black boxes. By isolating non-linearities within a feature-extraction phase and enforcing a strict linear or additive combination at the decision boundary, these architectures achieve a powerful synergy: the representation capacity of deep learning, bound to the mathematical truthfulness of a linear model.