To understand why a deep neural network made a specific prediction, explainable AI (XAI) frequently relies on a comforting simplification: linearity.
Popular interpretability frameworks often operate on the assumption that even if a deep network is globally complex, its behavior around a specific input can be approximated by a simple, additive combination of features. We see this when we assign independent "importance scores" to pixels in an image, words in a sentence, or internal concepts in a bottleneck layer. In an additive world, if Feature $A$ has an importance of $+3$ and Feature $B$ has an importance of $-2$, their combined presence yields a net effect of $+1$.
This assumption makes explanations highly readable for humans, but it introduces a fundamental mismatch. Deep neural networks derive their power precisely from non-linear feature interactions—where the joint effect of two features is radically different from the sum of their individual parts. When we force a linear lens onto a non-linear system, our explanations neglect the underlying curvature of the decision boundary, leading to incomplete or fundamentally inaccurate interpretations.
The Failure Modes of Local Linearity
The most common approach to local explanation is to fit a linear surrogate model (like a ridge regression) to the model's outputs in a small neighborhood around an input. However, as Jianbo Chen et al. demonstrate in Learning to Explain: An Information-Theoretic Perspective on Model Interpretation, these local linear approximations routinely fail when a model's decision is driven by intricate, non-linear feature interactions.
Consider a medical diagnosis model predicting cancer risk. As noted by Jonathan C. Garneau et al. in Feature Gradient Flow for Interpreting Deep Neural Networks in Head and Neck Cancer Prediction, using linear approximations to explain fundamentally non-linear clinical models oversimplifies the model's logic. If two clinical risk factors only become dangerous when present simultaneously, a linear attribution method will distribute credit evenly between them or miss the interaction entirely, obscuring the true logic driving high-performance predictions.
This blind spot manifests across diverse domains:
- Computer Vision: In early network layers, scene factors like lighting, pose, and geometry are deeply entangled. Mathieu Aubry et al. show in Understanding Deep Features with Computer-Generated Imagery that linear variance decomposition fails to capture how these factors interact non-linearly before reaching higher levels of abstraction.
- Large Language Models: When analyzing why an LLM refuses a prompt, interpretability tools often assume features act independently. However, Nirmalendu Prakash et al. reveal in Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusal that critical safety and refusal behaviors are driven by complex, non-linear interactions between latent features that cannot be unraveled by additive assumptions.
- Transformer Mechanics: Transformers are often conceptualized as a linear sum of attention heads and MLP components. Yet, Daking Rai et al. show in Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones that this additive view fails to capture non-linear layer interdependencies, such as when noisy internal mechanisms interfere with and overshadow sound ones during syntactic tasks.
The Interpretability-Interaction Tradeoff
The tension between human readability and model fidelity creates a sharp tradeoff when discovering internal representations.
For instance, Concept Bottleneck Models (CBMs) restrict a network's architecture so that it must first predict human-understandable concepts (e.g., "has wings", "has feathers") before predicting a final class. To keep the reasoning transparent, the final step is typically a linear combination of these concepts.
As Andong Tan et al. argue in Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Concepts, this linear constraint prioritizes readability at the expense of accuracy; it cannot represent cases where concepts interact non-linearly to form a conclusion. Léo Andéol et al. echo this in A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation, noting that decomposing neural activations into a simple linear combination of concepts oversimplifies the true feature space and mischaracterizes model behavior.
Linear Assumption (Additive):
Concept A (Wings) -------> [+2.1] ---\
Concept B (Beak) -------> [+1.5] ----+---> Class: Bird
Concept C (Fur) -------> [-3.0] ---/
*Misses conditional logic (e.g., Wings + Beak amplify each other synergistically)*
Non-Linear Interaction (Polynomial/Joint):
Concept A (Wings) ---\___ Interaction Term [A × B] ---> [+5.2] ---\
Concept B (Beak) ---/ +---> Class: Bird
Concept C (Fur) -------------------------------------> [-3.0] ---/
Furthermore, when we try to discover these concepts or project representations to find out "why" a model chose one class over another, we are often bottlenecked by linear tools. Alon Jacovi et al. point out in Contrastive Explanations for Model Interpretability that projection-based interpretability methods rely heavily on the assumption that the final decision layer is linear, preventing them from capturing deeper, non-linear feature interactions.
When we enforce linear mapping during representation learning, we risk missing critical higher-order interactions within deep networks, a limitation highlighted by Navita Goyal et al. in Causal Differentiating Concepts: Interpreting LM Behavior via Causal Representation Learning. This limitation extends to text-based vision debugging; as Angus Nicolson et al. note in TextCAVs: Debugging vision models using text, linear activation vectors are often restricted to the penultimate layer because extending them to deeper, highly non-linear layers remains an open challenge.
Mathematical Frameworks for Non-Linear Interaction
To bridge this gap, researchers are developing mathematical frameworks that explicitly model feature interdependencies without sacrificing interpretability. The core intuition is simple: if a linear model uses a first-order approximation ($y \approx \sum w_i x_i$), we can capture interactions by moving to higher-order polynomial expansions or non-additive cooperative game theory.
Polynomial Expansions and PolySHAP
Standard feature attribution methods like KernelSHAP estimate a feature's importance by fitting a weighted linear regression over masked versions of the input. In PolySHAP: Extending KernelSHAP with Interaction-Informed Polynomial Regression, Fabian Fumagalli et al. replace this linear surrogate with a higher-degree polynomial regression.
By adding cross-product terms ($x_i x_j$), the polynomial explicitly measures how the presence of one feature modifies the impact of another. The authors mathematically prove that "paired sampling"—a common heuristic used to stabilize Shapley estimates—is actually equivalent to fitting a second-order polynomial, providing a rigorous foundation for capturing feature interactions efficiently.
Taking this to the extreme, Tingxiong Xiao et al. introduce HOPE: High-order Polynomial Expansion of Black-box Neural Networks. Instead of sampling the model externally, they derive exact recurrence rules to compute high-order Taylor polynomials directly through standard network layers. This explicitly maps the decision boundary's curvature, providing local explanations that account for complex feature interactions while running thousands of times faster than sampling-based alternatives.
Non-Additive Measures
Another path forward comes from cooperative game theory. Standard linear aggregation assumes that the worth of a coalition of features is simply the sum of their individual worths. In Introduction to the Special Issue in Honor of Michel Grabisch, Stéphane González et al. discuss the application of non-additive measures (such as the Choquet or Sugeno integrals). These mathematical tools allow the importance of a feature subset to be greater or less than the sum of its parts, offering a rigorous way to model feature synergies and redundancies directly within predictive systems.
The Ambiguity of Ground Truth
As we move away from linear models, evaluating whether an explanation is "correct" becomes significantly harder.
In a purely linear model, the ground truth importance of a feature is unambiguous—it is exactly the model's coefficient for that feature. As Dominik Janzing et al. explain in Feature relevance quantification in explainable AI: A causal problem, empirical validation of XAI methods often relies on these linear baselines where ground truth is clean. Once complex interactions are present, a feature's contribution depends entirely on the context of other features, making the "ground truth" attribution ambiguous and highly sensitive to whether feature removal is treated as a statistical correlation or a causal intervention.
This challenge is compounded by feature redundancies. As Lukas Pfannschmidt et al. note in Feature Relevance Determination for Ordinal Regression in the Context of Feature Redundancies and Privileged Information, while we have strong frameworks for identifying all relevant features in linear models, extending this logic to non-linear kernels or deep architectures remains a difficult open challenge.
Without accounting for these non-linearities, our interpretability tools will continue to provide elegant, highly readable, but fundamentally incomplete stories about how deep networks reason. The shift toward polynomial expansions, non-additive measures, and causal formulations represents a necessary step toward explanations that match the true complexity of the models they seek to describe.
Key Foundational Work
Scalable Variational Bayesian Matrix Factorization with Side Information (2014) — contributed 2 concept elements to this direction.
Understanding Deep Features with Computer-Generated Imagery (2015) — contributed 1 concept elements to this direction.