Isolating Feature Interactions via Functional ANOVA

47 concepts38 papers1 proposal

Machine learning models are often criticized as black boxes. When a deep neural network or a gradient-boosted tree makes a prediction, it blends features together in highly non-linear ways. If we want to understand why a model made a specific decision, we often turn to feature attribution methods like SHAP or Partial Dependence Plots (PDP).

However, traditional attribution methods suffer from a fundamental flaw: they frequently conflate a feature's independent impact (its main effect) with its collaborative impact (its interactions with other features). To solve this, researchers use a classical statistical framework: Functional Analysis of Variance (Functional ANOVA or fANOVA). This approach mathematically carves a black-box function into isolated, non-overlapping pieces representing pure main effects and pure interaction effects.

The Core Intuition: Decomposing the Black Box

Imagine a model $f(X)$ that predicts medical risk based on two features: age ($x_1$) and blood pressure ($x_2$). In its raw form, $f(X)$ is an opaque landscape. Functional ANOVA breaks this landscape down into a sum of distinct, hierarchical components:

$$f(X) = f_0 + f_1(x_1) + f_2(x_2) + f_{12}(x_1, x_2)$$

In this decomposition:

  • $f_0$ is the global baseline (the average prediction across the entire data distribution).
  • $f_1(x_1)$ is the pure main effect of age alone.
  • $f_2(x_2)$ is the pure main effect of blood pressure alone.
  • $f_{12}(x_1, x_2)$ is the pure pairwise interaction effect—the extra risk that arises specifically from the combination of a certain age and blood pressure, which cannot be explained by either factor individually.

As shown early on by Grace Wahba et al., this structural decomposition transforms opaque, flexible models into interpretable frameworks. By isolating the influence of individual variables and their specific combinations, we can see exactly what drives a prediction.

Why It Works: The Necessity of Orthogonality

Why can't we just fit any arbitrary additive functions and call it a decomposition? The problem is identifiability. Without strict constraints, there are infinite ways to split a function. For example, you could add a constant or a linear trend to $f_1(x_1)$ and subtract it from $f_{12}(x_1, x_2)$, and the total sum $f(X)$ would remain exactly the same.

To make the decomposition unique and meaningful, fANOVA enforces a strict orthogonality constraint. Every higher-order interaction term must integrate to zero with respect to the probability distribution of any of its constituent variables:

$$\int f_{12}(x_1, x_2) d\mathbb{P}(x_1) = 0 \quad \text{and} \quad \int f_{12}(x_1, x_2) d\mathbb{P}(x_2) = 0$$

This zero-mean integral constraint ensures that $f_{12}$ captures only the pure synergy between $x_1$ and $x_2$. It is completely stripped of the individual main effects.

In kernel-based modeling, this is achieved through argumentwise kernel centering. As explained by David Ginsbourger et al., constructing multi-dimensional kernels from centered univariate kernels ensures that different orders of functional effects remain statistically independent. This mathematical framework guarantees that variance is uniquely apportioned.

Xiaoyu Lu et al. leveraged this by introducing the Orthogonal Additive Kernel (OAK), demonstrating that enforcing this zero-mean constraint allows complex datasets to be modeled using a few low-dimensional, highly interpretable terms without sacrificing predictive power. Without this strict orthogonality, as noted by Yosra Marnissi et al., main effects can erroneously capture variance that belongs to interaction terms, leading to misleading conclusions about variable importance.

Unifying Explainable AI and Game Theory

How does this mathematical decomposition help us explain existing black-box models? It turns out that many popular Explainable AI (XAI) tools are actually trying to approximate or slice the functional ANOVA space.

Fabian Fumagalli et al. provided a unified framework showing that methods like SHAP, PDP, and Permutation Feature Importance can be mapped directly onto a grid defined by functional decomposition and cooperative game theory. By breaking down a model into a hierarchical sum of individual feature effects and multi-way interactions, we can isolate pure effects from the complex dependencies that occur when multiple features vary simultaneously.

When we look at local explanations, methods like Prediction Difference Analysis (PredDiff), extended by Nils Strodthoff et al., use anchored functional decomposition to break down a model's output into isolated main and joint effects. This quantifies the super-additive impact of feature combinations, revealing exactly when the predictive power of a feature set exceeds the sum of its individual parts.

Similarly, David Köhler et al. demonstrated that applying stacked orthogonality to functional decomposition prevents information leakage into complex terms. This maximizes the amount of model behavior that can be explained through simple, intuitive univariate visualizations.

Overcoming the Combinatorial Explosion

If functional ANOVA is so powerful, why isn't it used everywhere? The bottleneck is computational. If an input space has $d$ features, the number of potential interaction terms scales exponentially as $2^d$. Computing the high-dimensional integrals required for exact orthogonal decomposition quickly becomes impossible.

Recent breakthroughs focus on bypassing this combinatorial explosion through clever algorithmic design:

  • Tree-Based Estimation: Jinyang Liu et al. introduced FastPD, an algorithm that efficiently estimates partial dependence functions for arbitrary feature subsets. This allows for a complete functional decomposition of tree ensembles in linear time relative to the number of observations, even when features are highly correlated.
  • Recursive Kernel Tricks: To discover non-linear interactions in high-dimensional settings like genomics, Raj Agrawal et al. developed the SKIM-FA kernel. By utilizing a recursive formulation, it performs functional ANOVA decomposition and variable selection in linear time relative to the number of covariates, avoiding the need to explicitly enumerate interaction terms.
  • Fast Shapley Attributions: For Gaussian Processes structured with a functional ANOVA kernel, Majid Mohammadi et al. showed that exact Shapley values can be computed in quadratic time ($O(d^2)$) instead of exponential time ($O(2^d)$), making mathematically rigorous local and global explanations highly practical.

Scaling to Deep and Generative Models

The principles of functional ANOVA are not restricted to shallow models or post-hoc explanations; they are increasingly being baked directly into deep architectures to enforce intrinsic interpretability.

For instance, Kaspar Märtens et al. embedded a functional ANOVA structure directly into the decoder of a Variational Autoencoder (VAE). By structuring the decoder as a sum of independent neural networks representing the marginal effects of latent variables, observed covariates, and their non-linear interactions, they allowed the model to explicitly attribute data variance to specific input sources or their combinations.

In the Bayesian deep learning realm, Seokhun Park et al. proposed a framework that treats the network architecture itself as a learnable parameter to automatically decompose complex, high-dimensional functions into high-order variable interactions. This allows the model to capture sophisticated data patterns that are typically missed by models restricted to low-order effects, significantly improving predictive accuracy on highly interactive datasets without sacrificing structural clarity.

Key Foundational Work

Structured Machine Learning for Soft Classification with Smoothing Spline ANOVA and Stacked Tuning, Testing, and Evaluation (1993) — contributed 2 concept elements to this direction.
Implicit Wiener Series for Higher-Order Image Analysis (2004) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_functional_anova_decomposition_6d544430,
  title = {functional ANOVA decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/functional-anova-decomposition/rdr_6d5444303058432e88dd6c372930ffb4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_functional_anova_decomposition_6d544430_v1,
  title = {functional ANOVA decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/functional-anova-decomposition/rdr_6d5444303058432e88dd6c372930ffb4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake