Low-Cost Representation for Restricted Boltzmann Machines

Son Ngoc Tran, Artur S. d'Avila GarcezIconip 2014 · DOI

TL;DR

This paper introduces a method to create a "low-cost" representation of Restricted Boltzmann Machines (RBMs) that significantly reduces memory requirements while largely preserving their feature learning capabilities. This compression is achieved by converting RBM weights into a three-valued matrix and a score vector, paving the way for deploying these powerful models in memory-constrained environments.

What Problem Does This Paper Address?

Restricted Boltzmann Machines (RBMs) are powerful generative models widely used for feature extraction, particularly in deep learning architectures like Deep Belief Networks. However, their inherent structural complexity leads to substantial memory footprints, limiting their application in resource-scarce scenarios such as embedded systems or mobile devices. The authors aim to bridge this gap by developing a method to compress RBMs, making them more memory-efficient without a drastic loss in their ability to learn meaningful features from data.

Methods

The core of the proposed method lies in transforming the RBM's weight matrix. Instead of using real-valued weights, the authors convert each weight $w_{ij}$ into a combination of a three-valued element $s_{ij} \in \{-1, 0, 1\}$ and a real-valued score $c_j$. This is achieved by minimizing the squared Euclidean distance between the original weight vector $w_j$ and the compressed representation $c_j s_j$. The process involves an iterative refinement of the score $c_j$ and the three-valued vector $s_j$. This low-cost representation can then be further compressed by removing zero-valued elements in $s_j$. The paper also highlights that this representation can be translated into a logic-like language, offering a more interpretable understanding of the learned features.

Datasets

The authors evaluated their method on several standard datasets:

  • MNIST: A widely used dataset for handwritten digit recognition.
  • TICC Handwritten Characters Dataset: Used for person's letter recognition.
  • YALE Face Dataset: Used for face recognition.

They specifically mention using subsets of MNIST (10,000 samples for training, 2000 for validation, and 10,000 for testing) and the full MNIST dataset (60,000 training samples).

Key Insights and Ideas

  • Memory Compression without Catastrophic Accuracy Loss: The most significant insight is that a substantial reduction in memory usage (up to 99%) can be achieved while maintaining classification accuracy that is nearly identical to, or only slightly lower than, the original RBM.
  • Three-Valued Representation: The novel idea of using a $\{-1, 0, 1\}$ representation for weights, combined with a score vector, is key to achieving high compression ratios. This is particularly effective because it allows for efficient bit encoding.
  • Logic-like Interpretability: The low-cost representation naturally lends itself to translation into confidence-logic rules, offering a more human-understandable insight into the features learned by the RBM.
  • Pruning Effectiveness: The study demonstrates that pruning the low-cost representation (removing low-scoring features) is as effective, if not more so, than pruning the original RBM in terms of maintaining accuracy while further reducing memory.

Key Results

  • Memory Savings: The paper reports memory savings of up to 98.7% for floating-point representations and potentially higher with further pruning. For a specific example with 784 visible and 500 hidden units, the low-cost representation required 816,000 bits compared to 25,088,000 bits for a double-precision RBM.
  • Accuracy Preservation: On the MNIST dataset, a 95% saving in memory was achieved with no significant drop in accuracy. In general, the performance of the low-cost representation was found to be "almost identical" or "acceptably lower" than that of the full RBMs across various datasets.
  • SVM Baseline Comparison: For MNIST and YALE datasets, the low-cost representation offered only a slight improvement over a baseline SVM trained on raw input data. However, for TICC and MNIST10K, feature extraction using the low-cost RBMs significantly outperformed the baseline SVM.

Limitations and Future Work

  • Limitations: The paper acknowledges that the low-cost representation might result in a slight decrease in accuracy compared to the full RBM, especially when aggressive pruning is applied. The performance improvement over a baseline SVM is dataset-dependent.
  • Future Work: The authors intend to explore hardware implementation details for the low-cost representation and investigate its application in real-world scenarios using devices with limited memory. They also suggest further research into the relationship between pruning and accuracy drop.

Summary updated

BibTeX
@inproceedings{tran2014low,
  author = {Son Ngoc Tran and Artur S. d'Avila Garcez},
  title = {{Low-Cost Representation for Restricted Boltzmann Machines}},
  booktitle = {Iconip},
  year = {2014},
  doi = {10.1007/978-3-319-12637-1\_9},
}
Figure