Spectral Learning of Restricted Boltzmann Machines

Aurélien Decelle, Giancarlo Fissore, Cyril FurtlehnerarXiv 2017 · arXiv

Introduction

Restricted Boltzmann Machines (RBMs) are powerful generative models, yet their internal learning dynamics often remain a "black box." Training via Contrastive Divergence (CD) typically requires extensive hyperparameter tuning without a robust theoretical foundation. This paper addresses this gap by providing a framework to analyze the evolution of the weight matrix $W$ through its singular value decomposition (SVD). By tracking how features emerge from random noise, the authors characterize the transition from linear feature extraction to complex non-linear representation.

Theoretical Framework: Learning as Symmetry Breaking

The authors employ tools from statistical physics and Random Matrix Theory (RMT) to dissect the learning process. Initially, the weight matrix consists of random noise, with its eigenvalues following the Marchenko-Pastur distribution. Learning is viewed as a process of "symmetry breaking," where informative modes are pulled out of this random "bulk."

Spectral density of the weight matrix showing the emergence of structured eigenvalues (red) from the random bulk (blue) as they cross the linear threshold.

By projecting gradient ascent equations onto the basis of the weight matrix's singular vectors, the authors identify a critical variance threshold. Data modes with variance below a specific noise limit ($\sigma_v^2$) are damped, while those above it are amplified and eventually "escape" the bulk to represent structured data features.

The Learning Trajectory: From PCA to Compositional States

The learning process unfolds in two distinct regimes:

  1. The Linear Regime: In the early stages, the RBM behaves as a linear feature extractor. During this phase, the RBM's singular vectors align closely with the Principal Components (PCA) of the input data.
  2. The Non-Linear Transition: As singular values grow, the model enters a non-linear regime. Using mean-field theory and the "replica trick," the authors show that modes begin to deform and interact. They no longer represent independent components but rather "compositional states"—combinations of hidden units that describe specific clusters or multimodal distributions within the data (e.g., different stylistic variations of the same digit).

Key Results and Empirical Validation

Testing the theory on the MNIST dataset, the authors confirmed several key predictions:

  • Thresholding: Only data modes exceeding the predicted variance threshold were successfully learned by the model.
  • Spectral Evolution: The first modes learned by the RBM were indeed the top principal components of the handwritten digits.
  • Eigenvalue Flattening: In mature RBMs, the top of the singular value spectrum appears "flattened" compared to the raw data. This suggests the RBM assigns similar importance (free energy) to different configurations, which facilitates more effective generative sampling.
Evolution of the singular value spectrum ($w_\alpha$) compared to raw data, highlighting the thresholding effect and the eventual flattening of the top modes.

Limitations and Future Directions

While the framework provides significant clarity, it relies on the assumption of "Replica Symmetry." Future work is needed to explore the "spin glass" phase, where high disorder might lead to a breakdown of this symmetry. Additionally, while the current analysis simplifies the role of biases to maintain mathematical tractability, incorporating full bias dynamics remains a challenge for future research.

Summary updated

Abstract

The Restricted Boltzmann Machine (RBM), an important tool used in machine learning in particular for unsupervized learning tasks, is investigated from the perspective of its spectral properties. Starting from empirical observations, we propose a generic statistical ensemble for the weight matrix of the RBM and characterize its mean evolution. This let us show how in the linear regime, in which the RBM is found to operate at the beginning of the training, the statistical properties of the data drive the selection of the unstable modes of the weight matrix. A set of equations characterizing the non-linear regime is then derived, unveiling in some way how the selected modes interact in later stages of the learning procedure and defining a deterministic learning curve for the RBM.
BibTeX
@misc{decelle2017spectral,
  author = {Aurélien Decelle and Giancarlo Fissore and Cyril Furtlehner},
  title = {{Spectral Learning of Restricted Boltzmann Machines}},
  year = {2017},
  eprint = {1708.02917},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.02917},
}
Figure