Additive Gaussian Processes

David Duvenaud, Hannes Nickisch, Carl Edward Rasmussen

TL;DR

This paper introduces "Additive Gaussian Processes" (AGPs), a flexible model that generalizes standard Gaussian Processes and Generalized Additive Models. AGPs can automatically discover and quantify the importance of interactions between input variables, leading to improved interpretability and state-of-the-art performance in regression and classification tasks.

What Problem Does This Paper Address?

Traditional statistical models often fall into two camps: those that assume relationships depend on variables individually (like Generalized Additive Models or GAMs) and those that assume relationships depend on all variables simultaneously (like standard Gaussian Processes with squared-exponential kernels). While GAMs are interpretable, they struggle with complex interactions. Standard GPs are flexible but can be "black boxes," making it hard to understand how variables influence the outcome. The challenge is to create a model that offers both the interpretability of simpler models and the flexibility of complex ones, allowing us to understand which combinations of variables are important and to what degree.

Methods

The core innovation is the additive kernel. Instead of a single kernel function, the AGP kernel is a sum of terms representing interactions of increasing order. A first-order term models individual variable effects, a second-order term models pairs of variables, and so on, up to the full interaction term. This structure allows the model to learn the contribution of each interaction order.

Key methodological contributions include:

  • Efficient Kernel Evaluation: While the number of interaction terms can grow exponentially with the number of input dimensions, the authors develop an efficient O(D²) method (where D is the input dimension) to compute the kernel and its derivatives, making the model tractable.
  • Hyperparameterization: The model introduces "order variance" hyperparameters that directly quantify the variance attributed to each interaction order. This provides a clear interpretation of the model's structure.
  • Bayesian Hierarchical Kernel Learning (HKL) Analogy: The process of learning these order variances and individual base kernel parameters is framed as a form of Bayesian HKL.

Datasets

The paper evaluates AGPs on a variety of real-world datasets for both regression and classification tasks. These include:

  • Regression: concrete, pumadyn-8nh, servo, housing, pima, liver, heart.
  • Classification: breast, pima, sonar, ionosphere, liver, heart.

The authors also use a synthetic dataset with an "L-shaped" structure to demonstrate the model's ability to perform long-range inference.

Key Insights and Ideas

  • Interpretability through Interaction Orders: The "order variance" hyperparameters are a powerful tool. They allow us to see if the data is best explained by individual variable effects (like a GAM), pairwise interactions, or more complex combinations. This bridges the gap between interpretable and flexible models.
  • Discovering Non-local Structure: AGPs can capture relationships that extend beyond local interpolation. This is crucial for datasets where the underlying function has a structure that isn't confined to the immediate vicinity of training points.
  • Generalization Power: By being a superset of both GAMs (first-order interactions) and standard squared-exponential GPs (full interactions), AGPs can adapt to various underlying data structures. If the data is truly additive, it excels; if not, it can still learn complex relationships.
  • Automatic Relevance Determination (ARD) for Interactions: Similar to how ARD in standard GPs learns the importance of individual input dimensions, AGPs learn the importance of different orders of interaction.

Key Results

  • State-of-the-Art Performance: AGPs achieve state-of-the-art results on several regression and classification benchmarks, often outperforming standard GP models and other methods like HKL.
  • Competitive with Simpler Models: On datasets where low-order interactions dominate, AGPs perform comparably to or better than GAMs.
  • Robustness: The additive model never performs significantly worse than other models and sometimes performs significantly better.
  • Interpretability in Practice: Table 1 shows learned order variances on real datasets, demonstrating how the model identifies dominant interaction orders (e.g., high variance for 3rd order on the 'concrete' dataset, or 1st order on 'pima').

Limitations and Future Work

  • Computational Cost: While efficient evaluation methods are provided, the computational cost of evaluating the Gram matrix can still be O(N²DR) (where N is the number of data points, D is input dimension, and R is max interaction order). For very high dimensions or when considering all orders up to D, this can be significant. The authors suggest limiting the maximum interaction order if computational resources are constrained.
  • Hyperparameter Optimization: While the method optimizes kernel hyperparameters, the authors note that in some cases, the optimization might get stuck in local maxima, leading to slightly worse performance than SE-GP. They suggest that integrating out kernel hyperparameters could further improve performance.
  • Future Work: The paper implies that further exploration of the learned interaction structures and their implications for domain understanding is a natural next step. The model's ability to handle covariate-shift problems is also highlighted as a promising area.

Summary updated

BibTeX
@inproceedings{duvenaud2011additive,
  author = {David Duvenaud and Hannes Nickisch and Carl Edward Rasmussen},
  title = {{Additive Gaussian Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2011},
  url = {https://openreview.net/forum?id=B1ZC0LbO-S},
}

Loading related content...

Figure