Determining the optimal architecture for a neural network—specifically, the right number of hidden neurons—is traditionally a matter of trial and error, heuristic pruning, or cross-validation. While Bayesian methods are frequently used to regularize network weights, they often assume a fixed model dimension or rely on approximations that may not hold in practice.
This paper introduces a "Full Bayesian" framework for Radial Basis Function (RBF) neural networks that treats the number of neurons as a variable to be estimated alongside weights and noise parameters. By employing Reversible Jump Markov Chain Monte Carlo (RJMCMC) and providing a geometric convergence proof, the authors demonstrate a method that is highly accurate, parsimonious, and significantly more robust to prior settings than previous approaches.
Joint Model Selection and Parameter Estimation
The authors tackle the challenge of joint model selection and parameter estimation. Instead of merely finding the best weights for a fixed 10-neuron network, their system determines whether the network should have 8, 10, or 15 neurons in the first place. Crucially, the model is designed to be "robust," meaning its structural conclusions do not change wildly if initial assumptions about the data's noise or spread are slightly tweaked.
The RJMCMC Methodology
The core of the paper is the application of a Full Hierarchical Bayesian model combined with Reversible Jump MCMC (RJMCMC).
- Hierarchical Priors: Instead of fixing hyper-parameters (which control the distribution of the weights), the authors treat them as random variables with their own hyper-priors. This allows the model to automatically adapt to the data's scale and noise level.
- Reversible Jump MCMC: Standard MCMC operates in a fixed-dimensional space. RJMCMC, however, allows the simulation to "jump" between spaces of different dimensions. During the sampling process, the algorithm can propose adding a neuron (a "birth" step) or removing one (a "death" step).
- Analytical Integration: To make the algorithm efficient, the authors analytically integrate out "nuisance parameters"—specifically, the linear weights ($\alpha$) and the noise variance ($\sigma^2$). This reduces the search space to just the number of neurons ($k$) and their centers ($\mu$), significantly speeding up convergence.
- Convergence Theory: The authors provide a formal proof that their RJMCMC algorithm achieves geometric convergence, guaranteeing that the distribution of the Markov chain approaches the true posterior distribution at an exponential rate.
Application: The Robot Arm Dataset
The paper evaluates the method using the Robot Arm Data, a well-known machine learning benchmark originally introduced by David MacKay. The task involves mapping the joint angles of a robot arm ($x_1, x_2$) to its end-effector position ($y_1, y_2$). The setup utilizes 200 observations for training, 200 for testing, and employs cubic basis functions for the RBF network.
Key Findings and Results
- Exceptional Robustness: One of the most striking findings is the model's insensitivity to the choice of hyper-parameters (specifically $\beta_{\delta^2}$). Whether the prior spread was set to 0.1 or 100, the Mean Square Error (MSE) and the estimated number of neurons remained remarkably stable.
- High Accuracy: The MSE on the test set was approximately 0.00502 to 0.00505. The authors note this is about 10% better than previously reported results using other Bayesian or non-Bayesian methods on the same dataset.
- Parsimony (Occam's Razor): The RJMCMC approach naturally favors simpler models that explain the data well. The algorithm successfully converged on a consistent, parsimonious architecture, with histograms of the model order ($k$) showing a clear peak around 13 to 15 neurons for the robot arm data.
- The Power of Integration: By integrating out the weights and noise variance, the authors transformed a high-dimensional sampling problem into a much more manageable one, focusing the dimensional "jumps" strictly on the structural parameters of the network.
Limitations and Future Work
While highly effective, the approach has limitations that pave the way for future research:
- Computational Complexity: RJMCMC is computationally intensive compared to standard backpropagation, which may limit its use in extremely large-scale or real-time applications.
- Basis Function Selection: The current work assumes a fixed type of basis function (e.g., cubic). Future research could allow RJMCMC to jump between different types of basis functions (like sigmoidal vs. RBF) within the same model.
- Non-Stationary Environments: The paper focuses on static datasets. Developing a sequential version of the algorithm to handle environments where data distributions change over time is an ongoing goal.
- Variable Selection: Extending the RJMCMC framework to perform input variable selection—deciding which inputs $x$ are actually relevant to the prediction—represents a highly promising avenue for future development.
Summary updated