Efficient Learning of Deep Boltzmann Machines

Ruslan Salakhutdinov, Hugo LarochelleAISTATS 2010 · OpenReview

Abstract

We present a new approximate inference algorithm for Deep Boltzmann Machines (DBM’s), a generative model with many layers of hidden variables. The algorithm learns a separate “recognition” model that is used to quickly initialize, in a single bottom-up pass, the values of the latent variables in all hidden layers. We show that using such a recognition model, followed by a combined top-down and bottom-up pass, it is possible to efficiently learn a good generative model of high-dimensional highly-structured sensory input. We show that the additional computations required by incorporating a top-down feedback plays a critical role in the performance of a DBM, both as a generative and discriminative model. Moreover, inference is only at most three times slower compared to the approximate inference in a Deep Belief Network (DBN), making large-scale learning of DBM’s practical. Finally, we demonstrate that the DBM’s trained using the proposed approximate inference algorithm perform well compared to DBN’s and SVM’s on the MNIST handwritten digit, OCR English letters, and NORB visual object recognition tasks.

Introduction: The Challenge of Undirected Deep Models

In the landscape of unsupervised deep learning during the late 2000s, Deep Belief Networks (DBNs) were the dominant architecture. However, DBNs are hybrid models: they consist of a stack of directed layers with an undirected Restricted Boltzmann Machine (RBM) at the top. This structure limits information flow during initial inference to a purely bottom-up pass, lacking true top-down feedback to resolve ambiguities in lower-level features.

Deep Boltzmann Machines (DBMs) offer a fully undirected alternative where every connection is symmetric. This allows for bidirectional communication between layers, enabling high-level abstractions to refine low-level representations. Despite their theoretical advantages, DBMs historically suffered from slow inference. Standard variational inference requires solving complex mean-field fixed-point equations iteratively, often taking dozens of steps per gradient update. This computational bottleneck made DBMs impractical for large-scale tasks compared to the efficient single-pass inference of DBNs.

Comparison of the hybrid directed/undirected Deep Belief Network (left) and the fully undirected Deep Boltzmann Machine (right).

Amortized Inference via Recognition Models

The core contribution of this work is a "learning to do inference" strategy. Rather than initializing mean-field iterations from scratch for every input, the authors propose training a separate, deterministic recognition model. This model, essentially a feed-forward neural network, is trained to provide a "warm start" for the latent variables.

Methodology

The training process involves several key stages:

  1. Greedy Pre-training: The DBM is initialized by stacking RBMs trained layer-by-layer. This provides a strong starting point for the generative weights.
  2. Recognition Model Initialization: The recognition model's weights ($R^1, R^2, R^3$) are derived from these same pre-trained RBM weights, ensuring the fast bottom-up pass is aligned with the generative model's structure.
  3. Refined Inference: For any given input, the recognition model performs a single bottom-up pass to estimate the hidden states. This initial guess is then refined by a small, fixed number of mean-field iterations (e.g., $K=1$ or $5$).
  4. Joint Optimization: As the DBM's generative weights are updated to better model the data, the recognition model is simultaneously fine-tuned via backpropagation. It learns to minimize the divergence between its fast predictions and the more accurate refined states produced by the mean-field steps.
The greedy pre-training of RBMs (left) and the resulting DBM architecture (right) showing the recognition model's bottom-up weights.

This approach "amortizes" the cost of inference. By learning the mapping from input pixels to latent states, the system avoids re-solving the optimization problem from scratch for every new sample.

Experimental Evaluation

The proposed method was tested on three benchmarks: MNIST (handwritten digits), OCR Letters, and the NORB dataset (3D objects). NORB, in particular, served as a rigorous test for handling high-dimensional, structured sensory data.

Key Results

  • Significant Speedups: On the NORB dataset, the recognition model (MFRec-1) achieved a ~24x speedup in inference time compared to full mean-field convergence (0.69s vs 16.5s for 100 images). Total training time was reduced from 5 days to just 1 day.
  • Superior Performance: The DBM outperformed both DBNs and SVMs on the NORB task, achieving a 7.6% error rate (compared to 8.3% for DBNs and 11.6% for SVMs).
  • Generative Fidelity: Using Annealed Importance Sampling (AIS), the authors confirmed that DBMs trained with this fast approximate inference maintained high generative quality, with log-likelihoods comparable to models trained using expensive, full mean-field iterations.

Conclusion

By integrating a fast recognition model with undirected deep architectures, this research bridges the gap between the representational power of DBMs and the computational efficiency of DBNs. The results demonstrate that top-down feedback is not just a theoretical benefit but a practical necessity for high-performance generative and discriminative modeling. While modern architectures like VAEs have since evolved these concepts further, this work remains a foundational step in making deep undirected models viable for large-scale machine learning.

Summary updated

Abstract

We present a new approximate inference algorithm for Deep Boltzmann Machines (DBM’s), a generative model with many layers of hidden variables. The algorithm learns a separate “recognition” model that is used to quickly initialize, in a single bottom-up pass, the values of the latent variables in all hidden layers. We show that using such a recognition model, followed by a combined top-down and bottom-up pass, it is possible to efficiently learn a good generative model of high-dimensional highly-structured sensory input. We show that the additional computations required by incorporating a top-down feedback plays a critical role in the performance of a DBM, both as a generative and discriminative model. Moreover, inference is only at most three times slower compared to the approximate inference in a Deep Belief Network (DBN), making large-scale learning of DBM’s practical. Finally, we demonstrate that the DBM’s trained using the proposed approximate inference algorithm perform well compared to DBN’s and SVM’s on the MNIST handwritten digit, OCR English letters, and NORB visual object recognition tasks.
BibTeX
@inproceedings{salakhutdinov2010efficient,
  author = {Ruslan Salakhutdinov and Hugo Larochelle},
  title = {{Efficient Learning of Deep Boltzmann Machines}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2010},
  url = {https://openreview.net/forum?id=xwPUm2HCiq},
}
Figure