Multi-Kernel Gaussian Processes

Arman Melkumyan, Fabio RamosIJCAI 2011 · OpenReview

Abstract

Multi-task learning remains a difficult yet important problem in machine learning. In Gaussian processes the main challenge is the definition of valid kernels (covariance functions) able to capture the relationships between different tasks. This paper presents a novel methodology to construct valid multi-task covariance functions (Mercer kernels) for Gaussian processes allowing for a combination of kernels with different forms. The method is based on Fourier analysis and is general for arbitrary stationary covariance functions. Analytical solutions for cross covariance terms between popular forms are provided including Matern, squared exponential and sparse covariance functions. Experiments are conducted with both artificial and real datasets demonstrating the benefits of the approach.

Introduction

Gaussian Processes (GPs) are a powerful tool for Bayesian regression, yet extending them to multi-task learning—where multiple related outputs are predicted simultaneously—presents significant challenges. The primary hurdle lies in defining "valid" cross-covariance functions that describe the relationships between different tasks while maintaining positive semi-definiteness.

Historically, the standard approach (co-kriging) has relied on using the same kernel type for every task. However, real-world data is rarely uniform; one task might be characterized by smooth transitions, while another exhibits high volatility. Forcing these heterogeneous tasks to share a single kernel type often leads to sub-optimal modeling. This paper introduces a framework that allows different tasks to employ entirely different kernel types, such as mixing smooth Squared Exponential (SqExp) kernels with more rugged Matérn kernels.

Methodology

The proposed methodology is built upon the convolution process. Any stationary covariance function $k(x, x')$ can be represented as the convolution of a basis function $g(u)$ with itself. The authors utilize Fourier analysis to derive these basis functions: by transforming a kernel into the frequency domain, calculating its square root, and performing an inverse transform, the underlying $g(u)$ can be identified for most stationary kernels.

Once basis functions $g_i$ and $g_j$ are determined for two different tasks, their cross-covariance is defined as the convolution of $g_i$ and $g_j$. The paper provides analytical solutions for several combinations, including SqExp $\times$ Matérn 3/2 and combinations involving sparse kernels.

Comparison of independent GPs, standard multi-task GPs, and the proposed multi-kernel GPs on simulated data.

As shown in Figure 1, the multi-kernel approach (c) better captures the varying smoothness of different tasks compared to independent GPs (a) or standard multi-task GPs restricted to a single kernel type (b).

Experimental Evaluation

The framework was evaluated on four datasets: 1D simulated data, an Iron Ore dataset, the Jura geostatistics benchmark, and a Concrete Slump dataset. In every experiment, the "Multi-Kernel" GP outperformed the standard "Single-Kernel" multi-task GP.

In the Jura dataset, which involves predicting concentrations of cadmium (Cd), lead (Pb), and zinc (Zn) in soil, the multi-kernel approach demonstrated superior performance. By allowing different minerals to use different kernels, the model achieved lower Standardized Mean Squared Error (SMSE) across all targets.

Performance comparison on the Jura dataset using various kernel combinations.

As seen in Table 4, the best-performing configuration used a Matérn 3/2 kernel for Cadmium and Squared Exponential kernels for Lead and Zinc, yielding the lowest error across the board. Similar improvements were observed in the Iron Ore dataset, where the combination of Matérn 3/2 (for iron and silica) and Squared Exponential (for alumina) achieved an absolute error of 2.2293, significantly lower than the 2.7995 achieved by using Squared Exponential for everything.

Discussion and Conclusion

The core insight of this work is that heterogeneity in multi-task learning is a feature to be embraced rather than a constraint to be smoothed over. By allowing tasks to maintain their individual characteristics (e.g., smoothness vs. roughness) while still sharing information through derived cross-covariance terms, the multi-kernel GP achieves higher predictive accuracy.

Furthermore, the use of Fourier transforms provides a general "recipe" for creating new cross-kernels, moving the field beyond trial-and-error derivations. While currently limited to stationary kernels, this framework sets the stage for future extensions into non-stationary modeling and large-scale applications via sparse kernel formulations.

Summary updated

Abstract

Multi-task learning remains a difficult yet important problem in machine learning. In Gaussian processes the main challenge is the definition of valid kernels (covariance functions) able to capture the relationships between different tasks. This paper presents a novel methodology to construct valid multi-task covariance functions (Mercer kernels) for Gaussian processes allowing for a combination of kernels with different forms. The method is based on Fourier analysis and is general for arbitrary stationary covariance functions. Analytical solutions for cross covariance terms between popular forms are provided including Matern, squared exponential and sparse covariance functions. Experiments are conducted with both artificial and real datasets demonstrating the benefits of the approach.
BibTeX
@inproceedings{melkumyan2011multi,
  author = {Arman Melkumyan and Fabio Ramos},
  title = {{Multi-Kernel Gaussian Processes}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2011},
  url = {https://openreview.net/forum?id=Sy4t6XMO-r},
}
Figure