In the world of machine learning, practitioners often face a frustrating trade-off. On one hand, Generalized Additive Models (GAMs) are beautifully interpretable and easy to fit, but they struggle to capture complex interactions between variables. On the other hand, Gaussian Processes (GPs) utilizing squared-exponential kernels are incredibly flexible and powerful, but they act as "black boxes" that often fail to generalize when data is sparse or high-dimensional.
In Additive Gaussian Processes, Duvenaud, Nickisch, and Rasmussen introduce a framework that bridges this gap. Their model offers the interpretability of additive models alongside the predictive power of full-kernel GPs.
The Extremes of Regression Models
Most regression models fall into two extremes. Simple models like linear regression or GAMs assume that the output is just a sum of independent effects from each input variable ($y = f(x_1) + f(x_2) + \dots$). These are easy to plot and understand but miss "synergies" between variables—for example, the price of a house depends on both its size and its location in a way that isn't strictly additive.
Conversely, full-kernel GPs (like the standard SE-GP) allow every variable to interact with every other variable simultaneously. While powerful, these models are "local"—they only make good predictions near where they have seen data. In high-dimensional spaces, data is almost always sparse, leading to the "curse of dimensionality." Furthermore, it is notoriously difficult to "look inside" a standard GP to see which variables are driving the predictions.
The Additive GP Approach
The authors propose a Gaussian Process model that decomposes functions into sums of low-dimensional interactions of all orders. The core idea is to define a kernel that is a sum of kernels representing interactions of different complexities:
- 1st order: Effects of single variables (like a GAM).
- 2nd order: Effects of pairs of variables.
- ...up to D-th order: Interactions between all $D$ variables (like a standard SE-GP).
The challenge is that for $D$ inputs, there are $2^D$ possible interaction terms. Summing these naively is computationally impossible for even moderate dimensions.
The Innovation: The authors recognize that the sum of all $n$-th order interactions is mathematically equivalent to the $n$-th elementary symmetric polynomial. By applying the Newton-Girard formulae—a recursive mathematical method—they can compute the sum of all these exponential interactions in just $O(D^2)$ time.
They also introduce "order variance" hyperparameters ($\sigma^2_n$). By training the model, the GP automatically learns how much weight to give to 1st-order effects versus higher-order effects.
Key Insights and Interpretability
The authors evaluated their model on a variety of standard regression and classification benchmarks from the UCI repository, yielding several compelling insights:
- Automatic Complexity Discovery: The model doesn't force a specific level of interaction. On the
concretedataset, it discovered that 70% of the variance is explained by 1st-order terms. On theliverdataset, it found that nearly 100% of the variance comes from 3rd-order interactions. - Non-Local Generalization: Because the model can focus on low-order interactions, it can extrapolate along axes. If it learns a 1st-order effect for $x_1$, it can predict what happens when $x_1$ changes even if it hasn't seen that specific combination of $(x_1, x_2, \dots, x_D)$ before.
- Interpretability by Design: If the model learns that low-order interactions dominate, researchers can literally plot the 1D or 2D functions it has learned. This allows practitioners to verify if the model's underlying "logic" makes physical sense.
Results and Limitations
The Additive GP consistently matched or outperformed the standard SE-GP and Hierarchical Kernel Learning (HKL) across all tested regression tasks. Crucially, the model never performed significantly worse than the standard GP, proving that it can gracefully "collapse" back into a standard GP if the data genuinely requires high-order interactions. Despite considering an exponential number of terms, the $O(D^2)$ overhead for the kernel computation was negligible compared to the $O(N^3)$ cost of standard GP matrix inversion.
However, the approach is not without limitations:
- Optimization Challenges: Like all GPs with many hyperparameters, the marginal likelihood surface can have local maxima. Integrating out hyperparameters (e.g., using MCMC) could further improve performance.
- Computational Scaling: While $O(D^2)$ is efficient, for extremely high-dimensional feature spaces (thousands of inputs), this could still become a bottleneck.
- Maximum Interaction Limit: In their experiments, the authors capped the interaction order at 10 for speed, though the theory allows going up to $D$. Future work could explore if very high-order interactions ever provide meaningful gains in real-world data.
Ultimately, Additive Gaussian Processes offer a powerful, tractable way to achieve state-of-the-art predictive power without sacrificing the interpretability that is so often required in real-world applications.
Summary updated