Lazy Gaussian Process Committee for Real-Time Online Regression

Han Xiao, Claudia EckertAAAI 2013 · DOI

Gaussian Processes (GPs) are a cornerstone of Bayesian machine learning, prized for their ability to provide both accurate predictions and well-calibrated uncertainty estimates. However, their practical application in real-time, streaming environments has long been hindered by their computational complexity. Standard GP regression scales at $O(N^3)$, making it prohibitively expensive as data accumulates. This paper, presented at AAAI 2013, introduces the Lazy Gaussian Process Committee (LGPC), a framework designed to overcome these scaling hurdles while maintaining high predictive performance.

The Challenge of Online Gaussian Processes

In an online or streaming setting, data arrives sequentially, requiring the model to update and predict in milliseconds. GPs face two primary obstacles in this context:

  1. Computational Cost: Re-inverting the full covariance matrix at every step is impossible for large-scale streaming data.
  2. Hyperparameter Instability: The standard practice of re-optimizing hyperparameters (e.g., noise levels, length scales) with every new data point is not only slow but can cause "jittery" updates that degrade the smoothness of predictions.

The authors specifically target high-frequency scenarios like mouse-trajectory prediction for anomaly detection in online banking, where latency must be minimal.

The LGPC Framework

The Lazy Gaussian Process Committee addresses these issues through three core innovations:

1. Committee Structure and "Lazy" Learning

Instead of a single global GP, LGPC partitions data among a committee of $Q$ independent GPs. The "lazy" aspect is the most significant departure from standard methods: hyperparameters for each GP in the committee are randomly initialized and never updated during the online phase. This bypasses the most computationally taxing part of GP training. The diversity of the committee compensates for the lack of individual optimization.

2. Submodular Subset Selection

When a new data point arrives, the model does not update the entire committee. Instead, it employs submodular optimization to greedily select a small subset ($S$) of GPs that would benefit most from the new information. Because the selection criterion (log pseudo-likelihood) exhibits diminishing returns, a simple greedy algorithm provides a near-optimal selection very efficiently.

3. Incremental Updates and Sparsification

Each GP maintains its state using partitioned matrix inversion, allowing updates in $O(T^2)$ time, where $T$ is a small, fixed maximum number of points per GP. To prevent memory bloat, a sparsification step removes the least informative points based on mutual information scores.

Experimental Results

The LGPC was evaluated against standard benchmarks (e.g., houses, sarcos) and a custom mouse-trajectory dataset involving real-time tracking of users performing banking tasks.

Computational Efficiency

LGPC demonstrates superior scaling compared to standard GPR and other online variants. In tests with 8,000 training points, LGPC achieved significantly lower training and prediction times than standard GPR and Sparse Online GP (SOGP).

Training and prediction time comparison across various GP methods as the number of points increases.

As shown in the figure above, while standard GPR (blue squares) and LoGP (green crosses) show steep exponential growth in time, LGPC (dark blue circles) maintains a much flatter trajectory, enabling true real-time performance.

Real-Time Application: Mouse Tracking

In the mouse-trajectory task, the model had to predict $(x, y)$ coordinates in real-time to detect anomalies. While offline GPR often produced noisy or unusable results due to the complexity of the movements (especially when users interacted with virtual keyboards), LGPC successfully tracked these fluctuating trajectories with high fidelity.

LGPC's real-time prediction of a complex mouse trajectory (U3) during an online banking task.

Conclusion

The Lazy Gaussian Process Committee proves that "diversity trumps optimization" in online settings. By combining a committee of unoptimized but diverse GPs with efficient submodular selection, LGPC achieves a balance of speed and accuracy that was previously difficult to attain for Bayesian regression. Future work may explore dynamic committee sizing and more sophisticated initialization strategies to further refine the model's robustness.

Summary updated

BibTeX
@inproceedings{xiao2013lazy,
  author = {Han Xiao and Claudia Eckert},
  title = {{Lazy Gaussian Process Committee for Real-Time Online Regression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2013},
  doi = {10.1609/aaai.v27i1.8572},
}
Figure