Imagine you are interpreting a machine learning model that predicts a patient's risk of heart disease. The model looks at two features: Age and Smoking Status. If the model simply adds the risk of being older to the risk of smoking, it is an additive model.
But real life—and modern machine learning—is rarely additive. A young smoker might face a moderate risk, and an elderly non-smoker might face a moderate risk, but an elderly smoker faces a catastrophic risk that is far greater than the sum of its parts. This extra risk is a higher-order feature interaction. It is a synergy where the joint presence of multiple features creates an effect that cannot be explained by looking at any single feature in isolation.
While deep neural networks and tree ensembles excel at capturing these non-linear dependencies, explaining how they use them remains a critical challenge in machine learning interpretability. Traditional post-hoc explanation frameworks often suffer from a fundamental blind spot: they compress these multi-variable synergies into flat, individual importance scores, obscuring the true decision-making logic of the model.
The Illusion of Additivity
Most popular feature attribution methods operate under a simplifying assumption of additivity. They try to slice a model's prediction into a clean sum of individual feature contributions. As noted by Scott M Lundberg et al. in their foundational work on unified local attribution (A Unified Approach to Interpreting Model Predictions), focusing primarily on first-order feature contributions fails to capture complex interaction effects.
When an explanation method forces a highly non-linear model into an additive framework, it creates a misleading picture. Sebastian Bordt demonstrates in From Shapley Values to Generalized Additive Models and back that traditional additive techniques end up smearing complex, high-order interactions across individual feature scores. For instance, if features $A$ and $B$ only cause an effect when combined, an additive explanation might split the credit and assign half to $A$ and half to $B$. A practitioner looking at the resulting dashboard would assume $A$ and $B$ are independently important, completely missing the interdependent logic required by the model.
This limitation is widespread across different architectures and modalities:
- Recurrent Networks: Even when architectures are designed for interpretability by isolating hidden states for distinct variables, the mechanisms for explaining how joint multi-variable interactions drive specific outcomes over time remain underdeveloped, as shown by Tao Lin et al. (Exploring interpretable LSTM neural networks over multi-variable data).
- Computer Vision: In visual explanations, standard saliency maps often overlook how different image regions modulate one another. Wojciech Samek et al. emphasize in Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications that identifying these mutual modulations is critical for understanding non-linear domains. Furthermore, Zihao Li et al. note that simplified scoring functions in visual explanations fail to isolate the contributions of specific interacting feature clusters (FD-CAM: Improving Faithfulness and Discriminability of Visual Explanation for CNNs).
The Credit Assignment Problem for Synergy
To move beyond additive approximations, we must solve a core theoretical challenge: How do we mathematically isolate and distribute the credit of an interaction?
If a model's output changes because of a high-order interaction, we can model this using a Taylor expansion. As Huiqi Deng et al. establish in A Unified Taylor Framework for Revisiting Attribution Methods, a faithful explanation requires the correct distribution or isolation of these high-order interactive terms. If an explanation method misassigns these terms, it merely captures local sensitivity rather than the model's true decision-making process.
Additive Attribution: [Feature A Score] + [Feature B Score] (Synergy is smeared or lost)
Interaction Attribution: [Main Effect A] + [Main Effect B] + [Interaction AB Score]
To formalize this, researchers have converged on three major mathematical toolkits:
1. Cooperative Game Theory (Shapley Extensions)
In standard cooperative game theory, the Shapley value calculates a feature's average marginal contribution across all possible feature subsets. To capture interactions, researchers extend this to frameworks like Shapley Taylor values or $n$-Shapley values, which treat combinations of features as distinct players in a game.
However, a major hurdle is determining a principled mechanism for distributing the credit of a multi-feature interaction among the individual participating features. As analyzed by Daniel Lundstrom et al. using "synergy functions," different explanation methods vary widely in how they weight features based on their specific functional influence within a shared interaction (A Unifying Framework to the Analysis of Interaction Methods using Synergy Functions).
2. Functional ANOVA (fANOVA)
Derived from statistics, Functional ANOVA decomposes a function into a sum of orthogonal components: main effects for single variables, two-way interactions for pairs, and higher-order interactions for larger subsets. Fabian Fumagalli et al. provide a theoretical bridge in Unifying Feature-Based Explanations with Functional ANOVA and Cooperative Game Theory, mapping out how popular explainability tools handle interaction complexity and showing that explicitly separating individual contributions from joint interactions prevents features from appearing falsely unimportant in isolation.
3. Variance-Based Sensitivity Analysis
Instead of tracking marginal contributions across discrete subsets, we can measure how input perturbations affect the variance of the model's output. Thomas FEL et al. demonstrate that leveraging Sobol indices allows black-box explanation methods to efficiently capture higher-order interactions between distinct input regions, offering a more complete picture of model behavior than purely local perturbation methods (Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis).
Taming the Combinatorial Explosion
If higher-order attribution is mathematically superior, why isn't it the default? The answer lies in computational complexity.
To calculate the exact first-order Shapley values for a model with $d$ features, you must evaluate $2^d$ feature subsets. To calculate higher-order interactions involving three or more features, the search space explodes exponentially. For deep neural networks or high-dimensional datasets, exact computation becomes impossible.
Number of Features (d) --> Subsets to evaluate (2^d) --> Exponential Explosion
To make interaction-aware attribution practical, recent research focuses on two acceleration strategies:
- Algorithmic Optimizations for Specific Model Classes: For certain model families, structural properties can be exploited to bypass exponential sampling. For instance, Majid Mohammadi et al. introduced recursive combinatorial techniques that compute exact Shapley values for product-kernel methods in polynomial time, though extending these exact methods to higher-order structures like Shapley Taylor values remains an active challenge (Computing Exact Shapley Values in Polynomial Time for Product-Kernel Methods). For general models, Wangxuan Fan et al. propose framing attribution as a stochastic optimization task with momentum to accelerate convergence, though adapting such frameworks to handle multi-feature dependency structures is still an open problem (SIM-Shapley: A Stable and Computationally Efficient Approach to Shapley Value Approximation).
- Pruning and Selection Rules: Instead of evaluating all possible feature combinations, we can dynamically isolate the interactions that matter most. The authors of Accelerating Shapley Explanation via Contributive Cooperator Selection (Fan Yang et al.) introduce a Shapley chain rule to theoretically identify and select only the most "contributive" feature interactions, drastically reducing the required number of model evaluations while maintaining explanation faithfulness.
Why Disentangling Interactions Matters
Developing robust, interaction-aware attribution isn't just a theoretical exercise; it is a prerequisite for deploying machine learning safely in complex, high-stakes domains.
Scientific Inference and Biology
In scientific discovery, we use models to uncover real-world relationships. As Fiona Katharina Ewald et al. note, current feature importance tools are better suited for individual attribution than for disentangling complex dependencies, which limits our ability to conduct precise scientific inference (A Guide to Feature Importance Methods for Scientific Inference). This is especially true in genomics. Bing Bing Zhou et al. point out that low-ranked individual genes often drive critical biological mechanisms through complex, non-linear interactions (A multi-filter enhanced genetic ensemble system for gene selection and sample classification of microarray data). Similarly, standard feature selection methods can fail to distinguish between classes separated by high-order relationships (Hong Zhao et al., Hierarchical Feature Selection with Recursive Regularization).
Medicine and Biomarker Discovery
In clinical applications, understanding feature cooperation is vital for patient care. When analyzing brain scans for Autism Spectrum Disorder (ASD), Nicha C Dvornek et al. emphasize that aggregating interactions into a single score obscures how distinct brain regions cooperate, creating a clear need for techniques that can disentangle interaction scores to reveal precise mechanisms of synergy (Efficient Interpretation of Deep Learning Models Using Graph Structure and Cooperative Game Theory: Application to ASD Biomarker Discovery).
Recommender Systems
In industrial systems, such as ad-click prediction, models rely heavily on feature combinations (e.g., User Age $\times$ Time of Day $\times$ Ad Category). Michael Tsang et al. demonstrate that detecting these global interactions and explicitly encoding them can significantly boost performance, but precisely quantifying the individual contribution of each specific interaction remains an open challenge (Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction Detection).
Moving Forward
As the field matures, researchers are working to unify these disparate perspectives. Frameworks are being developed to provide uncertainty-aware, non-linear variable selection with rigorous theoretical guarantees (Wenying Deng et al., Towards a Unified Framework for Uncertainty-aware Nonlinear Variable Selection with Theoretical Guarantees).
Concurrently, the benchmarking of these algorithms is becoming more rigorous. As Mohamed Karim Belaid et al. argue in Compare-xAI: Toward Unifying Functional Testing Methods for Post-hoc XAI Algorithms into a Multi-dimensional Benchmark, the ability to test how explanation algorithms account for complex interactions is currently underdeveloped. Creating functional unit tests that can isolate and evaluate high-order dependencies within model explanations remains a vital frontier in Explainable AI.
Key Foundational Work
Product-based Neural Networks for User Response Prediction (2016) — contributed 1 concept elements to this direction.
A multi-filter enhanced genetic ensemble system for gene selection and sample classification of microarray data (2010) — contributed 1 concept elements to this direction.