Kernel Carpentry for Online Regression Using Randomly Varying Coefficient Model

Narayanan Unny Edakunni, Stefan Schaal, Sethu VijayakumarIJCAI 2007 · OpenReview

Abstract

We present a Bayesian formulation of locally weighted learning (LWL) using the novel concept of a randomly varying coefficient model. Based on this, we propose a mechanism for multivariate non-linear regression using spatially localised linear models that learns completely independent of each other, uses only local information and adapts the local model complexity in a data driven fashion. We derive online updates for the model parameters based on variational Bayesian EM. The evaluation of the proposed algorithm against other state-of-the-art methods reveal the excellent, robust generalization performance beside surprisingly efficient time and space complexity properties. This paper, for the first time, brings together the computational efficiency and the adaptability of 'non-competitive' locally weighted learning schemes and the modelling guarantees of the Bayesian formulation.

Introduction

In the domain of real-time robotics and high-dimensional sensing, regression algorithms must be both computationally efficient and highly adaptive. Locally Weighted Learning (LWL) and its derivative, Locally Weighted Projection Regression (LWPR), are widely used because they decompose complex global non-linear problems into a set of simpler local linear models. However, these methods often suffer from two significant drawbacks: the requirement for manual tuning of hyperparameters (such as learning rates and distance metrics) and a lack of formal probabilistic grounding. This lack of grounding complicates principled model selection and the handling of heteroscedastic noise—noise that varies across the input space.

This paper introduces a generative probabilistic model that retains the "independent local expert" structure of LWL while leveraging Bayesian machinery to automate parameter tuning and provide principled confidence measures.

The Randomly Varying Coefficient (RVC) Model

The core of the proposed approach is the Randomly Varying Coefficient (RVC) model. Unlike traditional regression where a single fixed vector is learned, the RVC model assumes each data point $i$ has its own regression coefficient $\beta_i$, drawn from a Gaussian distribution centered around a local mean.

Graphical model of the RVC framework showing the dependencies between variables.

Spatial Localization and Learning

Spatial localization is achieved by making the variance of the coefficient distribution dependent on the distance from the local model's center. Points near the center have low variance, enforcing linearity, while distant points have high variance, allowing the model to effectively ignore them.

To handle the intractability of the posterior distribution, the authors employ Variational Bayesian EM to derive efficient update rules. These updates are formulated recursively for online learning, enabling the model to process data points sequentially without storing historical data. Predictions are made using a "Product of Experts" approach, effectively weighting each local model by its self-reported confidence.

Experimental Evaluation

The RVC model was evaluated against state-of-the-art methods, including Gaussian Processes (GP) and LWPR, across several benchmarks such as the sinc function and real-world datasets like the UCI Boston Housing and Ozone datasets.

Comparison of RVC and GP performance on the sinc function, including adaptive bandwidth visualization.

Results and Insights

  • Convergence and Accuracy: RVC demonstrated convergence speeds significantly faster than LWPR on the sinc function. Furthermore, RVC achieved generalization performance comparable to Gaussian Processes. While GPs are often the benchmark for Bayesian regression, their $O(N^3)$ scaling is prohibitive for large datasets; RVC provides similar accuracy with $O(dM)$ scaling.
  • Automatic Bandwidth Selection: By placing a Gamma prior on bandwidth parameters, the model automatically determines the optimal size of local regions based on data density and complexity, as shown in the bandwidth plot of the evaluation figure.
  • Robustness: On the Boston Housing and Ozone datasets, RVC matched or slightly outperformed both LWPR and GPs, proving its robustness across different data distributions and dimensionalities.

Conclusion

The RVC model successfully bridges the gap between the efficiency of locally weighted learning and the rigor of Bayesian inference. It provides a scalable, online regression framework that handles varying noise levels and tunes its own complexity. Future research will focus on developing fully Bayesian methods for structural adaptation (adding/deleting models) and exploring full covariance matrices to better capture correlations between input dimensions.

Summary updated

Abstract

We present a Bayesian formulation of locally weighted learning (LWL) using the novel concept of a randomly varying coefficient model. Based on this, we propose a mechanism for multivariate non-linear regression using spatially localised linear models that learns completely independent of each other, uses only local information and adapts the local model complexity in a data driven fashion. We derive online updates for the model parameters based on variational Bayesian EM. The evaluation of the proposed algorithm against other state-of-the-art methods reveal the excellent, robust generalization performance beside surprisingly efficient time and space complexity properties. This paper, for the first time, brings together the computational efficiency and the adaptability of 'non-competitive' locally weighted learning schemes and the modelling guarantees of the Bayesian formulation.
BibTeX
@inproceedings{edakunni2007kernel,
  author = {Narayanan Unny Edakunni and Stefan Schaal and Sethu Vijayakumar},
  title = {{Kernel Carpentry for Online Regression Using Randomly Varying Coefficient Model}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2007},
  url = {https://openreview.net/forum?id=B1EKWHG_Wr},
}
Figure