Introduction
Restricted Boltzmann Machines (RBMs) are powerful generative models, yet their internal learning dynamics often remain a "black box." Training via Contrastive Divergence (CD) typically requires extensive hyperparameter tuning without a robust theoretical foundation. This paper addresses this gap by providing a framework to analyze the evolution of the weight matrix $W$ through its singular value decomposition (SVD). By tracking how features emerge from random noise, the authors characterize the transition from linear feature extraction to complex non-linear representation.
Theoretical Framework: Learning as Symmetry Breaking
The authors employ tools from statistical physics and Random Matrix Theory (RMT) to dissect the learning process. Initially, the weight matrix consists of random noise, with its eigenvalues following the Marchenko-Pastur distribution. Learning is viewed as a process of "symmetry breaking," where informative modes are pulled out of this random "bulk."
By projecting gradient ascent equations onto the basis of the weight matrix's singular vectors, the authors identify a critical variance threshold. Data modes with variance below a specific noise limit ($\sigma_v^2$) are damped, while those above it are amplified and eventually "escape" the bulk to represent structured data features.
The Learning Trajectory: From PCA to Compositional States
The learning process unfolds in two distinct regimes:
- The Linear Regime: In the early stages, the RBM behaves as a linear feature extractor. During this phase, the RBM's singular vectors align closely with the Principal Components (PCA) of the input data.
- The Non-Linear Transition: As singular values grow, the model enters a non-linear regime. Using mean-field theory and the "replica trick," the authors show that modes begin to deform and interact. They no longer represent independent components but rather "compositional states"—combinations of hidden units that describe specific clusters or multimodal distributions within the data (e.g., different stylistic variations of the same digit).
Key Results and Empirical Validation
Testing the theory on the MNIST dataset, the authors confirmed several key predictions:
- Thresholding: Only data modes exceeding the predicted variance threshold were successfully learned by the model.
- Spectral Evolution: The first modes learned by the RBM were indeed the top principal components of the handwritten digits.
- Eigenvalue Flattening: In mature RBMs, the top of the singular value spectrum appears "flattened" compared to the raw data. This suggests the RBM assigns similar importance (free energy) to different configurations, which facilitates more effective generative sampling.
Limitations and Future Directions
While the framework provides significant clarity, it relies on the assumption of "Replica Symmetry." Future work is needed to explore the "spin glass" phase, where high disorder might lead to a breakdown of this symmetry. Additionally, while the current analysis simplifies the role of biases to maintain mathematical tractability, incorporating full bias dynamics remains a challenge for future research.
Summary updated