Efficient Multioutput Gaussian Processes through Variational Inducing Kernels

Mauricio A. Álvarez, David Luengo, Michalis K. Titsias, Neil D. LawrenceAISTATS 2010 · OpenReview

Abstract

Multioutput Gaussian Processes (MOGPs) are essential for modeling correlated data across multiple outputs, such as sensor networks or financial markets. However, traditional MOGPs suffer from cubic computational scaling and struggle with non-smooth latent processes. This paper introduces Variational Inducing Kernels (VIKs), a framework that replaces discrete inducing points with continuous inducing functions. By leveraging a variational lower bound, the proposed method achieves linear scaling with respect to the number of data points while maintaining robustness against overfitting and erratic latent signals.

Introduction: The Challenge of Multioutput Modeling

Gaussian Processes (GPs) provide a flexible Bayesian framework for regression, but their application to multioutput scenarios is computationally demanding. Standard models often use Convolution Processes (CPs), where observed outputs are modeled as filtered versions of shared latent functions. While powerful, CPs scale as $O(N^3 D^3)$, where $N$ is the number of observations and $D$ is the number of outputs, making them impractical for large-scale datasets.

To address this, "sparse" approximations like PITC or FITC use a small set of "inducing points" to summarize the data. However, these methods rely on the assumption that latent functions are smooth. When the underlying process is highly discontinuous or "white noise," point-based summaries fail to capture the necessary information.

Smooth vs. Noisy Latent Functions: Point-based inducing sets (red/blue dots) struggle to represent highly erratic signals (b) compared to smooth ones (a).

Methodology: Variational Inducing Kernels (VIKs)

The authors propose a shift from inducing points to inducing functions. Instead of sampling the latent function at specific coordinates, they define inducing variables through a convolution: $$u = \int T(x, z) f(z) dz$$ where $T$ is a smoothing kernel. This "blurring lens" allows the model to capture the essence of even a white-noise latent process.

Variational Sparse Inference

The framework utilizes a variational approach to minimize the Kullback-Leibler (KL) divergence between the approximate and true posterior. This results in a rigorous lower bound on the marginal likelihood, which serves as an objective function for optimizing both model hyperparameters and the parameters of the inducing kernels. The resulting algorithm, DTCVAR, reduces computational complexity to $O(NDK^2)$, where $K$ is the number of inducing functions.

Key Results and Performance

The effectiveness of the VIK framework was demonstrated across several domains, including educational data (ILEA dataset), compiler performance optimization, and financial time series.

Comparison of Mean Absolute Error (MAE) across different training set sizes. DTCVAR and DTCVARS consistently outperform PITC and approach the performance of a Full GP.

Efficiency and Robustness

In the compiler performance dataset, DTCVAR achieved accuracy comparable to a Full GP while being 34 times faster (0.65 minutes vs. 22 minutes). Furthermore, in tasks involving missing data in financial markets, the model's ability to handle "white noise" latent forces (Stochastic Latent Force Models) allowed it to significantly outperform standard linear models, reducing the standardized mean square error from 0.3927 to 0.2795.

Conclusion

By introducing Variational Inducing Kernels, this work provides a scalable and theoretically sound method for Multioutput Gaussian Processes. The transition from inducing points to inducing functions enables the modeling of non-smooth physical processes and provides a natural defense against overfitting, paving the way for more robust physics-informed machine learning.

Summary updated

Abstract

Interest in multioutput kernel methods is increasing, whether under the guise of multitask learning, multisensor networks or structured output data. From the Gaussian process perspective a multiout...
BibTeX
@inproceedings{alvarez2010efficient,
  author = {Mauricio A. Álvarez and David Luengo and Michalis K. Titsias and Neil D. Lawrence},
  title = {{Efficient Multioutput Gaussian Processes through Variational Inducing Kernels}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2010},
  url = {https://openreview.net/forum?id=JCwo5dkm4yH},
}
Figure