PoRB-Nets: Poisson Process Radial Basis Function Networks

Beau Coker, Melanie Fernandes Pradier, Finale Doshi-VelezUAI 2020 · OpenReview

Abstract

Bayesian neural networks (BNNs) are flexible function priors well-suited to situations in which data are scarce and uncertainty must be quantified. Yet, common weight priors are able to encode little prior knowledge about functional properties. We introduce PoRB-Nets (Poisson Process Radial Basis Function Networks), which place a Poisson process prior over the centers of a radial basis function network. This allows for independent specification of a function's lengthscale and amplitude variance, bridging the gap between Gaussian Processes and BNNs.

Introduction

In Bayesian machine learning, the choice of "priors" over functions is critical for encoding expectations about smoothness and variance. Gaussian Processes (GPs) are favored for their interpretable hyperparameters but struggle with scalability and lack a parametric form. Conversely, Bayesian Neural Networks (BNNs) scale efficiently but suffer from opaque priors. A common issue in standard BNNs is non-stationarity: functions often appear "busier" near the origin and "flatter" elsewhere because hidden unit centers (defined by the ratio of biases to weights) tend to follow a Cauchy distribution, clustering near zero.

Comparison of function priors: BNNs exhibit non-stationarity (clustering near the origin), while PoRB-Nets maintain consistent uncertainty across the input space.

Methods

PoRB-Nets address these limitations by utilizing Radial Basis Function Networks (RBFNs) with a Poisson Process (PP) prior over the unit centers. This approach introduces several key innovations:

  1. Poisson Process Prior: The number and location of hidden units are determined by a Poisson process, allowing the architecture to adapt to data density.
  2. Decoupled Variance and Lengthscale: To prevent increased unit intensity from inflating function volatility, PoRB-Nets rescale the width of each basis function to be inversely proportional to the local intensity. This ensures the amplitude variance remains constant while the lengthscale (wiggliness) can vary.
  3. Inference via MCMC: The model employs Hamiltonian Monte Carlo (HMC) for continuous parameters and Birth-Death Metropolis-Hastings to dynamically adjust the number of hidden units.
  4. Non-Stationary Extension: By incorporating a Sigmoidal Gaussian Cox Process, PoRB-Nets can learn spatially varying smoothness, making them ideal for data with changing volatility.

Results and Discussion

The model was evaluated against synthetic GP-generated data, medical time-series (MIMIC-III), financial indices (VIX), and the classic motorcycle crash dataset.

Qualitative and Quantitative Performance

PoRB-Nets demonstrate "amplitude stationarity" by design, avoiding the inherent biases of standard BNNs. In non-stationary benchmarks like the motorcycle dataset, the model correctly identifies regions of high and low oscillation. Quantitatively, PoRB-Nets consistently match or exceed the test log-likelihood of GPs and BNNs across various benchmarks. Unlike standard BNNs, which are highly sensitive to the fixed number of hidden units, PoRB-Nets are robust due to their adaptive architecture.

Inference on a non-stationary function. The right panel shows how the inferred Poisson process intensity tracks the true varying lengthscale of the data.

Interpretable Intensity

A significant advantage of PoRB-Nets is the interpretability of the Poisson process intensity. The inferred intensity maps directly to the inverse lengthscale of the data, providing a clear visualization of where the underlying function is most complex. Furthermore, the paper establishes Hellinger consistency, providing a theoretical guarantee that the posterior converges to the true function as data increases.

Conclusion

PoRB-Nets offer a compelling middle ground between GPs and BNNs, providing the interpretability of the former with the parametric flexibility of the latter. While currently limited to single-hidden-layer architectures and potentially facing scaling challenges in very high dimensions, they represent a significant step toward more principled and interpretable Bayesian deep learning. Future work may explore multi-layer extensions and alternative activation functions.

Summary updated

Abstract

Bayesian neural networks (BNNs) are flexible function priors well-suited to situations in which data are scarce and uncertainty must be quantified. Yet, common weight priors are able to encode litt...
BibTeX
@inproceedings{coker2020porb,
  author = {Beau Coker and Melanie Fernandes Pradier and Finale Doshi-Velez},
  title = {{PoRB-Nets: Poisson Process Radial Basis Function Networks}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=xtkV1t_okYw},
}
Figure