Radial Basis Functions: A Bayesian Treatment

David Barber, Bernhard SchottkyNeurIPS 1997

Introduction

Radial Basis Function (RBF) networks are widely used for regression and classification tasks. When their centers are fixed, they operate as linear models that are straightforward to train. However, achieving optimal performance usually requires adapting these centers to the data—a process that transforms the RBF into a complex, non-linear model highly susceptible to overfitting.

Traditionally, researchers mitigate overfitting by tuning regularization hyperparameters via cross-validation. This paper introduces a fully Bayesian framework for RBF networks that moves beyond simple point estimates and computationally wasteful cross-validation. By employing a Kullback-Leibler (KL) divergence-based Gaussian approximation, the authors provide an analytic method to simultaneously optimize network parameters and automatically adapt regularization constants.

The Pitfalls of Traditional Training

Standard methods for training adaptive-center RBFs rely heavily on cross-validation, which presents two major drawbacks:

  1. Computational Waste: Cross-validation requires training the model multiple times across different data splits, only to discard most of the results in favor of a single "optimal" parameter estimate.
  2. Lack of Uncertainty Quantification: Standard point-estimate methods provide a single prediction without "error bars." This leaves users without a measure of confidence in the model's output, which is critical in real-world decision-making.

A Fully Bayesian Approach

To address these issues, the authors propose finding the posterior distribution of all parameters given the data, rather than settling for a single set of "best" weights and centers.

Key technical innovations in this approach include:

  • Ensemble Learning via KL Divergence: Instead of using the common Laplace approximation—which fits a Gaussian only at the peak (mode) of the distribution—the authors minimize the Kullback-Leibler (KL) divergence between a Gaussian approximation and the true posterior. This provides a "global" fit that is significantly more robust than the "local" Laplace method.
  • Analytic Solutions: A major breakthrough of this paper is demonstrating that for RBFs with fixed widths, the KL divergence is completely analytic. This enables a fast, iterative re-estimation formula for weights, centers, and regularization constants, avoiding the massive computational overhead of Markov Chain Monte Carlo (MCMC) sampling.
  • Gamma Priors: By assigning Gamma distributions to the hyperparameters (regularization and noise levels), the model can automatically "learn" the appropriate level of regularization directly from the training data.

Visualizing Posterior Fluctuations

To illustrate the Bayesian treatment, the authors explore a surface regression task. Unlike standard models that lock basis functions to specific points, the Bayesian approach maintains a distribution over the parameters.

Surface regression showing basis function center fluctuations

Figure 1 visualizes how basis function centers fluctuate under the posterior distribution. This information preservation allows the model to perform "model averaging," leading to a final prediction that is more nuanced than any single network configuration could achieve.

1D Regression Results and Key Insights

The framework's quantitative benefits are demonstrated using a synthetic 1D regression problem. The target function, $g(x) = (1 + x - 2x^2) \exp(-x^2)$, was modeled using 40 noisy training points and an RBF network with just $K=6$ basis functions.

Comparison of Minimum KL, Laplace, and non-Bayesian methods on a 1D regression task

The results (shown in Figure 2) highlight several key advantages of the Minimum KL approach:

  • Superior Error Bars: The KL method produces highly reliable standard error estimates (dotted lines). In contrast, the Laplace method tends to underestimate the local posterior mass, resulting in less accurate and sometimes overly large error bounds.
  • Parsimony: The Bayesian adaptive center method achieves high accuracy with very few basis functions ($K=6$). Non-Bayesian methods typically require placing a basis function on nearly every training point to achieve comparable flexibility.
  • Automatic Regularization: The KL model successfully "self-tunes" its regularization constants and noise estimates. It matches the predictive performance of exhaustive cross-validation but does so in a principled, integrated, and mathematically rigorous manner.

Limitations and Future Work

While powerful, the proposed framework has limitations. In this specific treatment, the widths of the Gaussian basis functions were kept fixed; adapting them would require further extensions. Additionally, the method assumes the posterior distribution can be well-approximated by a single Gaussian, which may fail in highly complex or multi-modal scenarios. Finally, while the 1D and surface demonstrations are clear, the paper leaves the scaling behavior for very high-dimensional input spaces or massive datasets as an area for future exploration.

Summary updated

BibTeX
@inproceedings{barber1997radial,
  author = {David Barber and Bernhard Schottky},
  title = {{Radial Basis Functions: A Bayesian Treatment}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}
Figure