Modeling Feature Interactions in Neural Network Explanations

23 concepts22 papers1 proposal

Imagine a sentiment analysis model reading the phrase "not bad". If an interpretability method evaluates the words "not" and "bad" independently, it faces a paradox. In isolation, "bad" carries a highly negative score, while "not" acts as a neutral modifier. An additive explanation might sum these isolated scores and conclude the phrase is negative, completely missing the fact that their combination flips the sentiment to positive.

This scenario illustrates the core flaw of the independent feature assumption in Explainable AI (XAI). Traditional attribution methods—such as vanilla saliency maps, standard LIME, or classic Shapley values—treat input features as a flat, independent list. They measure the marginal contribution of each feature in isolation. However, deep neural networks are inherently non-linear "interaction machines." By ignoring how features work together, traditional explanations suffer from a severe faithfulness gap.

The Blind Spot of Additive Attribution

When an interpretability framework treats input features as independent units, it fails to capture the holistic context required for accurate model reasoning. As Zifan Wang et al. formalize through the concepts of necessity and sufficiency, a feature's importance is rarely absolute; it is highly contingent on the presence or absence of other features.

When local explanation methods evaluate features independently, they omit critical feature interactions where the importance of one variable depends entirely on another (Farhad Shakerin et al.). In extreme scenarios like XOR-like logic, this independent evaluation breaks down entirely. As Nils Strodthoff et al. demonstrate, if a model's prediction relies entirely on the joint presence of multiple features, individual feature importance scores can drop to exactly zero. This phenomenon completely hides the true drivers of a model's decision-making process.

This reliance on marginal, additive attribution creates a fundamental faithfulness gap (Landon Butler et al.). It forces a highly non-linear model into an oversimplified linear narrative, leading to inaccurate attributions when variables are physically, statistically, or logically linked (Malte von Bloh et al.).

The Mathematics of Synergy: Quantifying Interactions

To resolve this limitation, a growing body of research focuses on explicitly modeling and quantifying higher-order feature interactions. Instead of asking, "How much did feature $A$ contribute?" these methods ask, "How much did the synergy between feature $A$ and feature $B$ contribute beyond the sum of their individual parts?"

Game-Theoretic Interactions

Standard Shapley values allocate a model's prediction across individual features. To capture synergy, researchers have extended this framework to Shapley Interaction Indices. Fabian Fumagalli et al. developed SHAP-IQ, a unified framework that efficiently approximates any-order Shapley interactions. This approach allows users to isolate the exact synergistic or antagonistic effects of arbitrary feature groups.

Because calculating exact game-theoretic interactions scales exponentially with the number of features, computational efficiency is a major hurdle. To make this practical, Patrick Kolpaczki et al. introduced SVARM-IQ, which uses stratified sampling to approximate high-order interactions in a fraction of the time. Similarly, Majid Mohammadi et al. frame supervised learning as a multilinear game solved via Support Vector Machines, bypassing the additive assumption to capture non-linear feature combinations directly.

Gradient and Hessian Extensions

Another way to capture interactions is through calculus. While standard saliency maps look at the first derivative (the gradient), interaction-aware methods look at the second derivative (the Hessian matrix). The cross-derivative $\frac{\partial^2 f}{\partial x_i \partial x_j}$ explicitly measures how the gradient of feature $i$ changes with respect to feature $j$.

Joseph D. Janizek et al. introduced Integrated Hessians, extending the popular Integrated Gradients framework to pairwise interactions. This architecture-agnostic approach provides a theoretically grounded way to reveal complex logic like linguistic negation.

First-Order (Gradient):   Measures isolated change of a single feature.
Second-Order (Hessian):   Measures how one feature's change alters another feature's importance.

However, computing the Hessian is notoriously expensive. Sahil Singla et al. proved a surprising theoretical boundary: for high-confidence predictions in models with many classes, the Hessian becomes approximately rank-one and parallel to the gradient. This means first-order methods are often "good enough" when a model is certain, but second-order interaction tracking becomes absolutely essential when the model is uncertain or highly sensitive.

Structuring the Interaction Space

Measuring raw interaction scores is only half the battle; presenting them in a human-readable format is equally critical. A flat list of pairwise interactions scales quadratically, quickly becoming overwhelming. Researchers use structured representations to make these interactions interpretable.

Hierarchies and Graphs

In natural language processing, words interact across long distances due to the self-attention mechanisms of Transformers. Yiming Ju et al. proposed a method that detects feature interactions to build hierarchical word groupings. By abandoning rigid adjacency rules, this approach maps how distant tokens interact to form unified semantic concepts.

For computer vision, evaluating pixels in isolation ignores structural context, such as adjacent pixels forming an edge. Wenbo Guo et al. address this by using "decoys"—subtly perturbed inputs—to aggregate saliency scores and preserve inter-feature dependencies. To visualize these relationships, Vivswan Shitole et al. introduced Structured Attention Graphs (SAGs), which use beam search to uncover multiple, independent image regions that form conjunctions to drive a prediction. Similarly, Kosuke Sumiyasu et al. developed MoXI, a framework that isolates collective pixel groups based on their joint game-theoretic interactions, achieving high visualization accuracy by focusing purely on synergistic regions.

Flat Saliency Map:       [Pixel A: 0.1]  [Pixel B: 0.2]  [Pixel C: 0.0]
Structured Graph:        [Pixel A <--> Pixel B] ===> Jointly triggers Object Detection

Argumentative and Logical Frameworks

Interactions can also be framed symbolically. Piyawat Lertvittayakumjorn introduced AXPLR, a method that models feature interactions as a graph of "attacks" and "supports" within a computational argumentation framework. Instead of a list of weights, the explanation reads like a structured debate.

However, mapping continuous model mathematics to discrete logic introduces a fundamental tension. Francesca Toni et al. demonstrate a strict tradeoff between correctness (faithfulness to the model's true math) and completeness (capturing every complex interaction). Resolving this requires non-monotonic logic refinements to handle cases where a feature's impact completely reverses based on its context.

Combinatorial Search and Counterfactual Paths

A final perspective on feature interactions comes from observing how a model behaves when groups of features are altered or removed simultaneously.

If you remove features one by one based on isolated importance scores, you will often miss the Minimal Feature Set required to change a model's prediction. Because the impact of a single feature depends on its context, Chen Qian et al. framed this as a combinatorial Knapsack Problem (CIDR), using dynamic refinement to account for synergistic relationships during feature removal.

This interdependence is equally vital for counterfactual explanations (explanations that show how to change the input to flip the prediction). Burouj Armgaan et al. note that standard one-shot explanation methods predict a joint mask for all features simultaneously, failing to model the marginal, step-by-step impact of individual changes. To capture these dependencies, Bastian Pfeifer et al. developed CPATH, an algorithm that tracks sequential counterfactual paths. By finding trajectories of feature changes rather than isolated perturbations, it naturally preserves the underlying correlations and interactions that drive complex black-box models.

Key Foundational Work

Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation (2019) — contributed 1 concept elements to this direction.
Induction of Non-Monotonic Logic Programs to Explain Boosted Tree Models Using LIME (2019) — contributed 1 concept elements to this direction.
Explaining Explanations: Axiomatic Feature Interactions for Deep Networks (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_feature_independence_assumption_feature__4cebdc65,
  title = {feature independence assumption (feature interaction neglect)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-independence-assumption-feature-interaction-neglect/rdr_4cebdc6509524500bf380fc10b256c28},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_feature_independence_assumption_feature__4cebdc65_v1,
  title = {feature independence assumption (feature interaction neglect)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-independence-assumption-feature-interaction-neglect/rdr_4cebdc6509524500bf380fc10b256c28/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake