Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey

Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, Mark CrowleyarXiv 2021 · OpenReview

Abstract

This is a tutorial and survey paper on Boltzmann Machine (BM), Restricted Boltzmann Machine (RBM), and Deep Belief Network (DBN). We start with the required background on probabilistic graphical models, Markov random field, Gibbs sampling, statistical physics, Ising model, and the Hopfield network. Then, we introduce the structures of BM and RBM. The conditional distributions of visible and hidden variables, Gibbs sampling in RBM for generating variables, training BM and RBM by maximum likelihood estimation, and contrastive divergence are explained. Then, we discuss different possible discrete and continuous distributions for the variables. We introduce conditional RBM and how it is trained. Finally, we explain deep belief network as a stack of RBM models. This paper on Boltzmann machines can be useful in various fields including data science, statistics, neural computation, and statistical physics.

Introduction: Breaking the Neural Network Winter

Before the mid-2000s, artificial intelligence faced a significant hurdle: the "neural network winter." While researchers theorized that deeper networks could model more complex data, they were practically untrainable. As layers were added, the gradients used for learning would shrink to near zero—a phenomenon known as the vanishing gradient problem—leaving the initial layers of the network essentially untrained.

This paper surveys the models that broke this deadlock. By treating neural networks as physical systems with "energy states," researchers found a way to initialize deep networks using unsupervised learning, providing a "good starting point" that allowed standard backpropagation to finally succeed in deep architectures.

From Physics to Energy-Based Modeling

The foundation of these models lies in Energy-Based Modeling, drawing directly from the Ising model in statistical physics. RBMs define a probability distribution over data based on an "energy function." The goal of training is to lower the energy of "real" data points and raise the energy of "fake" or noisy ones. This perspective views a neural network as a system of interacting particles where states with lower energy are more probable, according to the Boltzmann distribution.

The Restricted Architecture

Unlike a standard Boltzmann Machine where every neuron can connect to every other neuron, a Restricted Boltzmann Machine (RBM) is a bipartite graph. This specific restriction is what makes the model computationally tractable.

Comparison between (a) a general Boltzmann Machine with intra-layer connections and (b) a Restricted Boltzmann Machine (RBM) with a bipartite structure.

As shown in the figure above, the RBM (b) eliminates connections within a layer (visible-to-visible or hidden-to-hidden), allowing only connections between layers. This structure enables researchers to learn "latent representations" of data—essentially discovering the hidden structure of the input without requiring explicit labels.

Training via Contrastive Divergence

Calculating the exact gradient for these models requires a "partition function," which is an intractable sum over all possible states. To overcome this, the paper explains the Contrastive Divergence (CD) algorithm. CD uses Gibbs Sampling to approximate the gradient quickly.

The intuition behind CD is "negative sampling": the model looks at the data it is currently generating (its "wrong beliefs") and iteratively adjusts its weights to move away from those samples, effectively refining its internal representation of the true data distribution.

Deep Belief Networks and the 2006 Breakthrough

The paper details how RBMs are stacked to form a Deep Belief Network (DBN). In this architecture, each layer is trained one at a time in a "greedy" unsupervised fashion to represent the features of the layer below it.

This stacking technique, pioneered by Geoffrey Hinton in 2006, proved that unsupervised pre-training provides a superior initialization for weights compared to random assignment. This was the specific breakthrough that demonstrated deep networks could be trained, effectively ending the second neural network winter. Beyond classification, DBNs are generative models; they can "dream" or generate new data samples that resemble the training set, and they serve as powerful tools for dimensionality reduction.

Legacy and Modern Context

While DBNs were revolutionary, the authors acknowledge they have been largely superseded by modern innovations like ReLU activation functions, Dropout, and Batch Normalization, which allow for training deep networks from scratch. However, the principles of energy-based models continue to influence research in Deep Boltzmann Machines (DBMs) and complex generative modeling at the intersection of statistical mechanics and machine learning.

Summary updated

Abstract

This is a tutorial and survey paper on Boltzmann Machine (BM), Restricted Boltzmann Machine (RBM), and Deep Belief Network (DBN). We start with the required background on probabilistic graphical models, Markov random field, Gibbs sampling, statistical physics, Ising model, and the Hopfield network. Then, we introduce the structures of BM and RBM. The conditional distributions of visible and hidden variables, Gibbs sampling in RBM for generating variables, training BM and RBM by maximum likelihood estimation, and contrastive divergence are explained. Then, we discuss different possible discrete and continuous distributions for the variables. We introduce conditional RBM and how it is trained. Finally, we explain deep belief network as a stack of RBM models. This paper on Boltzmann machines can be useful in various fields including data science, statistics, neural computation, and statistical physics.
BibTeX
@misc{ghojogh2021restricted,
  author = {Benyamin Ghojogh and Ali Ghodsi and Fakhri Karray and Mark Crowley},
  title = {{Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey}},
  year = {2021},
  eprint = {2107.12521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.12521},
}
Figure