Overview
This paper provides a theoretical foundation for pretraining Deep Boltzmann Machines (DBMs) by demonstrating how the process improves the variational lower bound of the model. Building on this analysis, the authors introduce a novel pretraining algorithm that balances the "modeling work" more evenly across layers, leading to significantly better generative performance on complex datasets like MNIST and NORB.
The Challenge with Deep Boltzmann Machines
Deep Boltzmann Machines (DBMs) are powerful generative models consisting of multiple layers of undirected, symmetrically coupled hidden units.
Unlike Deep Belief Networks (DBNs), which use directed connections in their lower layers, DBMs are fully undirected. This structural difference makes DBMs excellent at capturing complex, non-linear dependencies, but it also makes them incredibly difficult to train.
The standard approach to training DBMs involves a "greedy" pretraining phase where layers are trained one by one as Restricted Boltzmann Machines (RBMs). While this heuristic—originally proposed by Salakhutdinov & Hinton in 2009—worked well in practice, it lacked the theoretical "guarantee of improvement" that exists for DBNs. Furthermore, the original method often forced the bottom-most layer to do most of the heavy lifting, leading to diminishing returns as more layers were added to the network.
A Balanced Pretraining Algorithm
The authors bridge the gap between DBN and DBM theory. They demonstrate that for a two-layer DBM, the standard pretraining procedure actually improves a variational lower bound on the log-probability of the data, but with a significant caveat: while DBN pretraining replaces the entire prior of a layer, DBM pretraining only replaces half of it.
To fix this imbalance, they propose a new method to control the amount of modeling work done by each layer.
- Replicated Hidden Units: They theoretically frame the DBM as having multiple sets of replicated hidden units. By choosing how many sets to replace during pretraining, they can dictate whether a layer should do 1/2, 2/3, or 1/3 of the modeling work.
- Asymmetric Weights: In practice, this is implemented by training RBMs with specific weight constraints. For example, to make the first layer do only 1/3 of the work, they train it with bottom-up weights that are three times larger than the top-down weights.
- Layer-by-Layer Load Balancing: They provide a concrete greedy algorithm for a 3-layer DBM that uses specific weight ratios (e.g., 3:1, 4:3) to ensure the modeling load is distributed equally across the hierarchy.
Key Insights
- The "Half-Prior" Realization: The most important insight is that standard DBM pretraining is inherently conservative. Because DBMs are undirected, the first hidden layer receives input from both the visible units below and the hidden units above. The old pretraining method only accounted for the lack of top-down feedback by doubling weights, which the authors show is mathematically equivalent to only replacing half the prior.
- Load Balancing: By spreading the modeling work equally (e.g., each layer doing 1/3 of the work in a 3-layer stack), the model avoids "bottlenecking" at the first layer.
- Theoretical Unity: The paper successfully places DBM pretraining on the same solid variational footing as DBNs, turning a successful heuristic into a principled algorithm.
Experimental Results
The authors evaluated their new pretraining strategy on two classic benchmarks: MNIST (handwritten digits) and NORB (3D toy objects photographed under various conditions). The new pretraining method consistently outperformed the original "weight-doubling" heuristic:
- MNIST (2-layer): The new "1/3-2/3" split achieved a test log-probability of -83.43, compared to -84.62 for the original method. Before any fine-tuning, the gap was even larger (~7 nats), showing the pretraining itself is much more effective.
- MNIST (3-layer): The balanced "1/3-1/3-1/3" model reached -83.02, outperforming the 2-layer version. Under the old pretraining method, 3-layer DBMs actually performed worse than 2-layer ones, proving that the old method failed to scale with depth.
- NORB: The new algorithm improved the variational bounds by at least 5 nats over standard DBMs, confirming that the benefits hold for more complex, high-dimensional data.
Limitations and Future Work
While the authors proved the variational bound improvement for two layers, they haven't yet extended the formal proof to three or more layers, though experimental results strongly suggest the logic holds. Additionally, training DBMs still requires computationally intensive MCMC-based stochastic approximation. Finally, the paper focuses on binary stochastic units; extending this balanced pretraining to Gaussian or other continuous-valued units (often used for raw image pixels) remains a natural next step for future research.
Summary updated