Stochastic Neural Networks with Monotonic Activation Functions
Lists
Only you see this
Abstract
We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences. This family, that we call exponential family RBM (Exp-RBM), is a subset of the exponential family Harmoniums that expresses family members through a choice of smooth monotonic non-linearity for each neuron. Using contrastive divergence along with our Gaussian approximation, we show that Exp-RBM can learn useful representations using novel stochastic units.
BibTeX
@inproceedings{ravanbakhsh2016stochastic,
author = {Siamak Ravanbakhsh and Barnabás Póczos and Jeff G. Schneider and Dale Schuurmans and Russell Greiner},
title = {{Stochastic Neural Networks with Monotonic Activation Functions}},
booktitle = {International Conference on Artificial Intelligence and Statistics},
year = {2016},
url = {https://openreview.net/forum?id=uF8QvT_ks_Y},
}| Rectified Linear Units Improve Restricted Boltzmann Machines | ICML · 2010 |
| A Probabilistic Framework for Nonlinearities in Stochastic Neural Networks | arXiv · 2017 |
| Annealing Gaussian into ReLU: a New Sampling Strategy for Leaky-ReLU RBM | arXiv · 2016 |
| Gaussian-Bernoulli RBMs Without Tears | arXiv · 2022 |
| Training Restricted Boltzmann Machines | Künstliche Intell · 2015 |
| An Infinite Restricted Boltzmann Machine | Neural Computation · 2016 |