Abstract
Bayesian neural networks (BNNs) are flexible function priors well-suited to situations in which data are scarce and uncertainty must be quantified. Yet, common weight priors are able to encode little prior knowledge about functional properties. We introduce PoRB-Nets (Poisson Process Radial Basis Function Networks), which place a Poisson process prior over the centers of a radial basis function network. This allows for independent specification of a function's lengthscale and amplitude variance, bridging the gap between Gaussian Processes and BNNs.
Introduction
In Bayesian machine learning, the choice of "priors" over functions is critical for encoding expectations about smoothness and variance. Gaussian Processes (GPs) are favored for their interpretable hyperparameters but struggle with scalability and lack a parametric form. Conversely, Bayesian Neural Networks (BNNs) scale efficiently but suffer from opaque priors. A common issue in standard BNNs is non-stationarity: functions often appear "busier" near the origin and "flatter" elsewhere because hidden unit centers (defined by the ratio of biases to weights) tend to follow a Cauchy distribution, clustering near zero.
Methods
PoRB-Nets address these limitations by utilizing Radial Basis Function Networks (RBFNs) with a Poisson Process (PP) prior over the unit centers. This approach introduces several key innovations:
- Poisson Process Prior: The number and location of hidden units are determined by a Poisson process, allowing the architecture to adapt to data density.
- Decoupled Variance and Lengthscale: To prevent increased unit intensity from inflating function volatility, PoRB-Nets rescale the width of each basis function to be inversely proportional to the local intensity. This ensures the amplitude variance remains constant while the lengthscale (wiggliness) can vary.
- Inference via MCMC: The model employs Hamiltonian Monte Carlo (HMC) for continuous parameters and Birth-Death Metropolis-Hastings to dynamically adjust the number of hidden units.
- Non-Stationary Extension: By incorporating a Sigmoidal Gaussian Cox Process, PoRB-Nets can learn spatially varying smoothness, making them ideal for data with changing volatility.
Results and Discussion
The model was evaluated against synthetic GP-generated data, medical time-series (MIMIC-III), financial indices (VIX), and the classic motorcycle crash dataset.
Qualitative and Quantitative Performance
PoRB-Nets demonstrate "amplitude stationarity" by design, avoiding the inherent biases of standard BNNs. In non-stationary benchmarks like the motorcycle dataset, the model correctly identifies regions of high and low oscillation. Quantitatively, PoRB-Nets consistently match or exceed the test log-likelihood of GPs and BNNs across various benchmarks. Unlike standard BNNs, which are highly sensitive to the fixed number of hidden units, PoRB-Nets are robust due to their adaptive architecture.
Interpretable Intensity
A significant advantage of PoRB-Nets is the interpretability of the Poisson process intensity. The inferred intensity maps directly to the inverse lengthscale of the data, providing a clear visualization of where the underlying function is most complex. Furthermore, the paper establishes Hellinger consistency, providing a theoretical guarantee that the posterior converges to the true function as data increases.
Conclusion
PoRB-Nets offer a compelling middle ground between GPs and BNNs, providing the interpretability of the former with the parametric flexibility of the latter. While currently limited to single-hidden-layer architectures and potentially facing scaling challenges in very high dimensions, they represent a significant step toward more principled and interpretable Bayesian deep learning. Future work may explore multi-layer extensions and alternative activation functions.
Summary updated