Additive Gaussian Processes

David K. Duvenaud, Hannes Nickisch, Carl Edward Rasmussen

In the world of machine learning, practitioners often face a frustrating trade-off. On one hand, Generalized Additive Models (GAMs) are beautifully interpretable and easy to fit, but they struggle to capture complex interactions between variables. On the other hand, Gaussian Processes (GPs) utilizing squared-exponential kernels are incredibly flexible and powerful, but they act as "black boxes" that often fail to generalize when data is sparse or high-dimensional.

In Additive Gaussian Processes, Duvenaud, Nickisch, and Rasmussen introduce a framework that bridges this gap. Their model offers the interpretability of additive models alongside the predictive power of full-kernel GPs.

The Extremes of Regression Models

Most regression models fall into two extremes. Simple models like linear regression or GAMs assume that the output is just a sum of independent effects from each input variable ($y = f(x_1) + f(x_2) + \dots$). These are easy to plot and understand but miss "synergies" between variables—for example, the price of a house depends on both its size and its location in a way that isn't strictly additive.

Conversely, full-kernel GPs (like the standard SE-GP) allow every variable to interact with every other variable simultaneously. While powerful, these models are "local"—they only make good predictions near where they have seen data. In high-dimensional spaces, data is almost always sparse, leading to the "curse of dimensionality." Furthermore, it is notoriously difficult to "look inside" a standard GP to see which variables are driving the predictions.

The Additive GP Approach

The authors propose a Gaussian Process model that decomposes functions into sums of low-dimensional interactions of all orders. The core idea is to define a kernel that is a sum of kernels representing interactions of different complexities:

  1. 1st order: Effects of single variables (like a GAM).
  2. 2nd order: Effects of pairs of variables.
  3. ...up to D-th order: Interactions between all $D$ variables (like a standard SE-GP).
Illustration of 1D kernels combining into 1st-order (additive) and 2nd-order (interactive) GP priors.

The challenge is that for $D$ inputs, there are $2^D$ possible interaction terms. Summing these naively is computationally impossible for even moderate dimensions.

The Innovation: The authors recognize that the sum of all $n$-th order interactions is mathematically equivalent to the $n$-th elementary symmetric polynomial. By applying the Newton-Girard formulae—a recursive mathematical method—they can compute the sum of all these exponential interactions in just $O(D^2)$ time.

They also introduce "order variance" hyperparameters ($\sigma^2_n$). By training the model, the GP automatically learns how much weight to give to 1st-order effects versus higher-order effects.

Key Insights and Interpretability

The authors evaluated their model on a variety of standard regression and classification benchmarks from the UCI repository, yielding several compelling insights:

  • Automatic Complexity Discovery: The model doesn't force a specific level of interaction. On the concrete dataset, it discovered that 70% of the variance is explained by 1st-order terms. On the liver dataset, it found that nearly 100% of the variance comes from 3rd-order interactions.
  • Non-Local Generalization: Because the model can focus on low-order interactions, it can extrapolate along axes. If it learns a 1st-order effect for $x_1$, it can predict what happens when $x_1$ changes even if it hasn't seen that specific combination of $(x_1, x_2, \dots, x_D)$ before.
  • Interpretability by Design: If the model learns that low-order interactions dominate, researchers can literally plot the 1D or 2D functions it has learned. This allows practitioners to verify if the model's underlying "logic" makes physical sense.
The Additive GP allows for plotting individual 1D and 2D interaction components, making the model highly interpretable.

Results and Limitations

The Additive GP consistently matched or outperformed the standard SE-GP and Hierarchical Kernel Learning (HKL) across all tested regression tasks. Crucially, the model never performed significantly worse than the standard GP, proving that it can gracefully "collapse" back into a standard GP if the data genuinely requires high-order interactions. Despite considering an exponential number of terms, the $O(D^2)$ overhead for the kernel computation was negligible compared to the $O(N^3)$ cost of standard GP matrix inversion.

However, the approach is not without limitations:

  • Optimization Challenges: Like all GPs with many hyperparameters, the marginal likelihood surface can have local maxima. Integrating out hyperparameters (e.g., using MCMC) could further improve performance.
  • Computational Scaling: While $O(D^2)$ is efficient, for extremely high-dimensional feature spaces (thousands of inputs), this could still become a bottleneck.
  • Maximum Interaction Limit: In their experiments, the authors capped the interaction order at 10 for speed, though the theory allows going up to $D$. Future work could explore if very high-order interactions ever provide meaningful gains in real-world data.

Ultimately, Additive Gaussian Processes offer a powerful, tractable way to achieve state-of-the-art predictive power without sacrificing the interpretability that is so often required in real-world applications.

Summary updated

Abstract

We introduce a Gaussian process model of functions which are additive. An additive function is one which decomposes into a sum of low-dimensional functions, each depending on only a subset of the input variables. Additive GPs generalize both Generalized Additive Models, and the standard GP models which use squared-exponential kernels. Hyperparameter learning in this model can be seen as Bayesian Hierarchical Kernel Learning (HKL). We introduce an expressive but tractable parameterization of the kernel function, which allows efficient evaluation of all input interaction terms, whose number is exponential in the input dimension. The additional structure discoverable by this model results in increased interpretability, as well as state-of-the-art predictive power in regression tasks.
BibTeX
@inproceedings{duvenaud2011additive,
  author = {David K. Duvenaud and Hannes Nickisch and Carl Edward Rasmussen},
  title = {{Additive Gaussian Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2011},
  url = {https://openreview.net/forum?id=B1ZC0LbO-S},
}
Figure