Introduction
Radial Basis Function (RBF) networks are prominent tools in the field of function approximation and interpolation, valued for their speed and structural simplicity. The training of an RBF network typically follows a three-stage pipeline: first, identifying the centers of the Gaussian kernels (often via clustering); second, determining the widths of these kernels; and finally, calculating the output weights through linear least squares.
While the optimization of centers and weights is well-documented, the determination of kernel widths often relies on crude heuristics. Practitioners frequently resort to setting a constant width for all kernels or calculating widths based solely on local nearest-neighbor distances. The former fails to account for non-uniform data distributions, while the latter adapts to density but ignores global function smoothness. This paper presents a systematic heuristic that balances local data density with global optimization to improve generalization and numerical stability.
Methodology: The Hybrid Scaling Heuristic
The proposed method introduces a two-part approach to width determination that unites local adaptation with global tuning.
- Local Base Widths: For each cluster $j$, the standard deviation ($\sigma_j^c$) of the training data points assigned to that cluster's Voronoi zone is calculated. This ensures the baseline width of each kernel reflects the local density of the input space.
- Global Scaling Factor: A global width scaling factor ($q$) is introduced, which is common to all kernels. The final width for each kernel is defined as $\sigma_j = q \times \sigma_j^c$.
- Grid Search Optimization: The optimal value for $q$ is determined by evaluating the network's Mean Square Error (MSE) on a validation set across a range of potential scaling factors.
The Optimization Landscape and Locality
A critical finding of this research is the behavior of the validation error as the scaling factor $q$ varies. Plotting MSE against $q$ frequently reveals multiple local minima.
The authors advocate for a strict selection rule: always choose the smallest $q$ that yields a minimum. This preference is rooted in the concept of "local decomposition." When $q$ is small, each Gaussian kernel models a specific local region of the input space. As $q$ increases, kernels begin to overlap excessively.
The Danger of Non-Local Decomposition
If a larger $q$ value (a subsequent minimum) is selected, the network enters a state of "non-local decomposition." In this regime, the network achieves low error not by modeling local features, but by using massive, opposing weights to cancel out the broad overlaps of the kernels.
As shown in the figure above, the weights can explode to magnitudes as high as $10^5$. This creates a highly brittle model where small numerical errors or slight changes in input can lead to wild fluctuations in output, rendering the model practically unusable despite a low theoretical MSE.
Experimental Results
The heuristic was validated across several benchmarks:
- 1D Sine Wave: The proposed method achieved an MSE of 0.0533, significantly outperforming Moody & Darken’s nearest-neighbor method (0.0844) and Haykin’s method (0.1334) while maintaining model stability.
- Complex 1D Function: While some traditional methods achieved lower raw MSE, they did so by sacrificing locality and incurring massive weights. The proposed heuristic successfully identified the optimal stable local minimum.
- Real-World Applications: The method proved effective on multi-dimensional tasks, including predicting water content in dried milk from spectral data and phosphene position prediction for medical implants. In both cases, the heuristic identified clear, stable scaling factors that generalized well to validation data.
Conclusion
By separating kernel width into a local density component and a globally optimized scaling factor, this approach provides a robust framework for RBF network configuration. It allows practitioners to achieve high accuracy while explicitly avoiding the numerical instabilities associated with non-local kernel decomposition. Future work may explore the integration of weight regularization (such as L2 penalties) to further penalize the weight explosions observed at high scaling factors.
Summary updated