Supervised Restricted Boltzmann Machines

Tu Dinh Nguyen, Dinh Q. Phung, Viet Huynh, Trung LeUAI 2017 · OpenReview

Abstract

Supervised Restricted Boltzmann Machines (sRBMs) provide a unified probabilistic framework for simultaneous representation learning and prediction. By integrating directed connections from hidden units to labels and utilizing a novel variational inference scheme, sRBMs overcome the limitations of traditional two-stage pipelines and existing supervised RBM variants. This paper explores the architecture, inference challenges, and empirical performance of sRBMs across classification and regression tasks.

Introduction

In the landscape of deep learning, Restricted Boltzmann Machines (RBMs) have long served as foundational tools for unsupervised feature learning. However, their application to supervised tasks—such as classification or regression—has historically relied on a disjointed two-stage pipeline. In this traditional approach, an RBM is first trained to extract features through data reconstruction; these features are then frozen and used as inputs for a separate classifier, such as a Support Vector Machine (SVM) or Multi-Layer Perceptron (MLP).

This methodology suffers from two primary drawbacks. First, the learned features are often suboptimal for the target task because the generative model never "sees" the labels during training. Second, the process introduces significant hyperparameter complexity, requiring the tuning of two independent models. While variants like the ClassRBM attempted to address this, their use of undirected links for labels limited their flexibility, particularly for regression tasks. The Supervised Restricted Boltzmann Machine (sRBM) addresses these issues by creating a hybrid model that retains generative power while incorporating a robust, directed discriminative component.

The sRBM Architecture

The sRBM is a hybrid probabilistic graphical model that fuses a generative component with a discriminative one. The generative part consists of a standard RBM with undirected connections between visible units ($v$) and hidden units ($h$). The discriminative part introduces directed connections from the hidden units ($h$) to the outcome variable ($y$).

Comparison of graphical models: (a) ClassRBM with undirected label links, (b) the proposed sRBM with directed links, and (c) the sRBM showing dependencies between hidden units.

By modeling the label $y$ as a "child" of the hidden units, the sRBM allows the labels to act as regularizers during the learning process. This forces the hidden units to learn representations that are not only good for reconstructing the input data but are also specifically tuned to distinguish between different classes or predict continuous values.

Scalable Variational Inference

The introduction of directed links from $h$ to $y$ creates a significant computational challenge known as the "explaining away" effect. In a standard RBM, hidden units are conditionally independent given the input. In the sRBM, however, the hidden units become dependent on each other when the label is known. This dependency makes standard Markov Chain Monte Carlo (MCMC) methods, such as Gibbs sampling, prohibitively slow.

To overcome this, the authors developed a scalable variational inference approach. By employing first-order and second-order Taylor series approximations to handle intractable log-partition functions, the sRBM can scale to large datasets. This mathematical framework also allows the model to handle any outcome variable within the exponential family, making it a versatile tool for both discrete classification and continuous regression.

Empirical Evaluation

The sRBM was evaluated against several benchmarks using diverse datasets, including MNIST (image classification), 20 Newsgroups (text classification), and CT Slices (regression).

Classification Performance

On the 20 Newsgroups dataset, the sRBM achieved a 23.2% error rate, significantly outperforming the state-of-the-art ClassRBM (24.9%). On the MNIST dataset, it delivered a competitive 2.21% error rate as a standalone model. Beyond raw accuracy, the quality of the learned representations was notably superior. As shown in t-SNE visualizations, the sRBM clusters data points by class much more cleanly than standard RBMs, indicating a more discriminative latent space.

t-SNE visualization of MNIST digits in the sRBM latent space, showing clear class separation.

Regression and Generative Capabilities

In the CT slice location task, the sRBM's Root Mean Square Error (RMSE) was significantly lower than both standard Ridge Regression and the traditional two-stage RBM+Ridge Regression pipeline. Furthermore, because the sRBM remains a generative model, it supports conditional generation. By "clamping" a specific label, the model can generate synthetic data points (e.g., images of a specific digit) that reflect the learned class characteristics.

Conclusion

The Supervised Restricted Boltzmann Machine successfully bridges the gap between generative and discriminative modeling. By allowing labels to guide feature extraction through a directed graphical structure and efficient variational inference, it produces more interpretable and task-specific representations. Future work may extend this framework to multi-task and multi-modal learning, further leveraging its ability to handle complex, interdependent data structures.

Summary updated

BibTeX
@inproceedings{nguyen2017supervised,
  author = {Tu Dinh Nguyen and Dinh Q. Phung and Viet Huynh and Trung Le},
  title = {{Supervised Restricted Boltzmann Machines}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2017},
  url = {https://openreview.net/forum?id=uxN1b_AT2bS},
}
Figure