Learning Class-relevant Features and Class-irrelevant Features via a Hybrid third-order RBM

Heng Luo, Ruimin Shen, Changyong Niu, Carsten UllrichAISTATS 2011

Abstract

Restricted Boltzmann Machines (RBMs) are powerful generative models, yet they often capture features that are irrelevant for classification tasks, such as background noise or stroke thickness. This paper introduces a Hybrid Third-order RBM architecture that explicitly separates hidden units into class-relevant and class-irrelevant groups. By leveraging the faster learning dynamics of shared features, the model forces class-specific units to focus on discriminative information. This approach provides a natural regularization mechanism, reducing the need for manual hyperparameter tuning and resulting in a more compact, efficient classifier. Evaluations on MNIST, NORB, and Caltech101 Silhouettes show significant improvements in classification accuracy and inference speed.

Introduction

Standard RBMs are typically trained to reconstruct input data in an unsupervised manner. While effective at feature detection, many learned features are "generative" rather than "discriminative." In tasks like digit recognition, an RBM might allocate significant capacity to modeling features common across all classes, which do not assist in distinguishing between specific labels. These irrelevant features can degrade classifier performance and increase computational costs. Traditionally, researchers use weight-decay strategies to mitigate this, but such methods are often difficult to tune and task-dependent.

Architecture and Methodology

The proposed Hybrid Third-order RBM architecture utilizes third-order connections where visible units and hidden units are gated by class labels. The hidden layer is split into two distinct types of units:

  1. Class-Irrelevant Units ($g$): These units are trained on all data to capture common statistical structures shared across classes (e.g., backgrounds or general shapes).
  2. Class-Relevant Units ($h$): These are partitioned into groups, each corresponding to a specific class label, learning only the features unique to that label.
Figure 1: The Hybrid Third-order RBM architecture, showing the separation of class-relevant units ($h$) and class-irrelevant units ($g$).

During training, class-irrelevant units receive updates from every example, allowing them to learn shared features rapidly. This "feature competition" forces class-relevant units to capture the remaining discriminative details. At test time, only the class-relevant units are used for classification, leading to a highly efficient and lightweight model.

Natural Regularization and Feature Competition

A significant advantage of this architecture is "natural regularization." Because class-irrelevant units handle the bulk of data reconstruction, the class-relevant units do not require large weight norms to represent the data. This behavior effectively replaces the need for manual weight-decay tuning.

Figure 6: Comparison of average weight norms for class-relevant and class-irrelevant units over the course of training.

As shown in Figure 6, class-irrelevant units maintain higher norms as they capture the primary generative components, while class-relevant units remain focused and compact. This temporal "race" during training ensures that shared features are captured by the "irrelevant" bucket first, preventing them from distracting the classifier.

Experimental Results

The model was benchmarked against several standard datasets:

  • MNIST: Achieved a test error of 1.04%, outperforming standard RBMs (3.39%) and Sparse HDRBMs (1.16%) while using roughly one-third the parameters for classification.
  • NORB: Recorded an error rate of 8.9%, surpassing SVMs (11.6%) and remaining competitive with more complex Deep Boltzmann Machines (DBMs), but with a much faster training and testing cycle.
  • Caltech101 Silhouettes: Reached a 28.6% error rate, a substantial improvement over previous benchmarks like Marlin’s result (34.2%).

Conclusion

The Hybrid Third-order RBM offers a robust framework for disentangling generative and discriminative features. While the third-order nature increases training complexity, the resulting efficiency at inference and the elimination of weight-decay tuning make it a compelling alternative for classification tasks. Future work may explore automated methods for determining the optimal ratio of hidden unit types and further refining discriminative fine-tuning to fully exploit the model's capacity.

Summary updated

BibTeX
@article{luo2011learning,
  author = {Heng Luo and Ruimin Shen and Changyong Niu and Carsten Ullrich},
  title = {{Learning Class-relevant Features and Class-irrelevant Features via a Hybrid third-order RBM}},
  journal = {AISTATS},
  year = {2011},
}

Loading related content...

Figure