Abstract
Multioutput Gaussian Processes (MOGPs) are essential for modeling correlated data across multiple outputs, such as sensor networks or financial markets. However, traditional MOGPs suffer from cubic computational scaling and struggle with non-smooth latent processes. This paper introduces Variational Inducing Kernels (VIKs), a framework that replaces discrete inducing points with continuous inducing functions. By leveraging a variational lower bound, the proposed method achieves linear scaling with respect to the number of data points while maintaining robustness against overfitting and erratic latent signals.
Introduction: The Challenge of Multioutput Modeling
Gaussian Processes (GPs) provide a flexible Bayesian framework for regression, but their application to multioutput scenarios is computationally demanding. Standard models often use Convolution Processes (CPs), where observed outputs are modeled as filtered versions of shared latent functions. While powerful, CPs scale as $O(N^3 D^3)$, where $N$ is the number of observations and $D$ is the number of outputs, making them impractical for large-scale datasets.
To address this, "sparse" approximations like PITC or FITC use a small set of "inducing points" to summarize the data. However, these methods rely on the assumption that latent functions are smooth. When the underlying process is highly discontinuous or "white noise," point-based summaries fail to capture the necessary information.
Methodology: Variational Inducing Kernels (VIKs)
The authors propose a shift from inducing points to inducing functions. Instead of sampling the latent function at specific coordinates, they define inducing variables through a convolution: $$u = \int T(x, z) f(z) dz$$ where $T$ is a smoothing kernel. This "blurring lens" allows the model to capture the essence of even a white-noise latent process.
Variational Sparse Inference
The framework utilizes a variational approach to minimize the Kullback-Leibler (KL) divergence between the approximate and true posterior. This results in a rigorous lower bound on the marginal likelihood, which serves as an objective function for optimizing both model hyperparameters and the parameters of the inducing kernels. The resulting algorithm, DTCVAR, reduces computational complexity to $O(NDK^2)$, where $K$ is the number of inducing functions.
Key Results and Performance
The effectiveness of the VIK framework was demonstrated across several domains, including educational data (ILEA dataset), compiler performance optimization, and financial time series.
Efficiency and Robustness
In the compiler performance dataset, DTCVAR achieved accuracy comparable to a Full GP while being 34 times faster (0.65 minutes vs. 22 minutes). Furthermore, in tasks involving missing data in financial markets, the model's ability to handle "white noise" latent forces (Stochastic Latent Force Models) allowed it to significantly outperform standard linear models, reducing the standardized mean square error from 0.3927 to 0.2795.
Conclusion
By introducing Variational Inducing Kernels, this work provides a scalable and theoretically sound method for Multioutput Gaussian Processes. The transition from inducing points to inducing functions enables the modeling of non-smooth physical processes and provides a natural defense against overfitting, paving the way for more robust physics-informed machine learning.
Summary updated