Robust and Scalable Gaussian Process Regression and Its Applications

Yifan Lu, Jiayi Ma, Leyuan Fang, Xin Tian, Junjun Jiang

Introduction

Gaussian Processes (GPs) are highly regarded non-parametric tools for regression due to their ability to provide principled uncertainty estimates. However, they possess a significant vulnerability: extreme sensitivity to outliers. Standard Gaussian Process Regression (GPR) assumes that noise follows a Normal distribution. Consequently, even a small number of outliers can disproportionately influence the regression curve, pulling it away from the true underlying signal.

This paper introduces a robust and scalable GPR framework designed to maintain accuracy even when up to 90% of the data consists of outliers. By integrating a mixture likelihood model with stochastic variational inference and sparse inducing variables, the authors present a solution that is both mathematically rigorous and computationally efficient enough for real-time applications.

Overview of robust GPR performance and applications. (a-c) Comparison of standard GPR vs. the proposed method under outlier noise. (d-e) Real-world applications in feature matching and spatial transcriptomics.

Methodology

The proposed framework rests on three methodological pillars designed to address the dual challenges of robustness and scalability:

  1. Mixture Likelihood Model: Rather than simply adopting heavy-tailed distributions (like Student-t), the authors explicitly model data as a mixture. Inliers follow a standard Gaussian distribution, while outliers are assumed to be sampled from a uniform distribution. This ensures that outliers are treated as independent of the GP, preventing them from distorting the learned function.
  2. Variational Inference (VI): The authors derive a variational lower bound (ELBO) to jointly infer the status of each data point (inlier vs. outlier) and the model hyperparameters. This transforms a complex integration problem into a tractable optimization task.
  3. Scalability via Inducing Variables and SVI: To overcome the $O(n^3)$ complexity of standard GPs, the model utilizes "inducing variables" to summarize the data, reducing complexity to $O(nm^2)$. Integrating Stochastic Variational Inference (SVI) further allows the model to process data in mini-batches, enabling it to scale to massive datasets.

Experimental Evaluation

The model was rigorously tested across synthetic and real-world benchmarks:

  • Synthetic Benchmarks: The 1D Neal and 10D Friedman datasets were used to evaluate accuracy under controlled outlier ratios reaching 80%.
  • Computer Vision: Evaluated on YFCC100M, CPC, and HPatches for tasks like relative pose and homography estimation. These tasks involve thousands of points with high outlier ratios.
  • Biological Imaging: The model was applied to spatial transcriptomics from mouse olfactory bulbs and human breast cancer biopsies for dense gene expression imputation.
Real-time feature matching using the proposed robust GPR framework, achieving high precision (0.964) and low latency (10.914ms).

Key Results and Applications

Superior Robustness and Precision

On the Neal dataset with 80% outliers, the proposed model achieved a Mean Absolute Error (MAE) of 0.03, significantly outperforming standard Variational Free Energy (VFE) methods which recorded an MAE of 0.76. In feature matching, the model reached a precision of 85.3% on the YFCC100M dataset, nearly doubling the performance of state-of-the-art deep learning methods like OANet (51.2%).

Real-Time Efficiency

A standout feature of the SVI implementation is its speed. The model can process 4,000 feature points in approximately 9.8 milliseconds. This efficiency makes it viable for high-speed computer vision pipelines where robust spatial consistency is required.

GP as a Motion Field

A creative insight from the paper is treating motion vectors between images as a GP. This allows the model to distinguish between spatially consistent "smooth" true matches and random, spurious outliers, effectively using the GP as a geometric filter.

Discussion and Future Work

While the uniform distribution for outliers proves highly effective for random noise, it may be a "weak assumption" in cases of structured, systematic sensor bias. Additionally, while the framework handles fixed hyperparameters well, optimizing kernel parameters in extremely high-dimensional spaces remains a challenge. Future research may explore deeper integration with neural networks (Deep GPs) to learn more complex feature representations while maintaining the robustness demonstrated here.

Summary updated

Abstract

This paper introduces a robust and scalable Gaussian process regression (GPR) model via variational learning. This enables the application of Gaussian processes to a wide range of real data, which are often large-scale and contaminated by outliers. Towards this end, we employ a mixture likelihood model where outliers are assumed to be sampled from a uniform distribution. We next derive a variational formulation that jointly infers the mode of data, i.e., inlier or outlier, as well as hyperparameters by maximizing a lower bound of the true log marginal likelihood. Compared to previous robust GPR, our formulation approximates the exact posterior distribution. The inducing variable approximation and stochastic variational inference are further introduced to our variational framework, extending our model to large-scale data. We apply our model to two challenging real-world applications, namely feature matching and dense gene expression imputation. Extensive experiments demonstrate the superiority of our model in terms of robustness and speed. Notably, when matching 4k feature points, its inference is completed in milliseconds with almost no false matches. The code is at github.com/YifanLu2000/Robust-Scalable-GPR.
BibTeX
@inproceedings{lu2023robust,
  author = {Yifan Lu and Jiayi Ma and Leyuan Fang and Xin Tian and Junjun Jiang},
  title = {{Robust and Scalable Gaussian Process Regression and Its Applications}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02102},
}
Figure