Restricted Boltzmann Machines With Gaussian Visible Units Guided by Pairwise Constraints

Jielei Chu, Hongjun Wang, Hua Meng, Peng Jin, Tianrui Li

Introduction

Restricted Boltzmann Machines (RBMs) have long been celebrated as powerful unsupervised feature extractors. However, in practical applications, they often suffer from two significant drawbacks: a lack of task-specific guidance and an inherent difficulty in handling real-valued data. Standard RBMs capture the statistical structure of data without prioritizing information relevant to downstream tasks like clustering. Furthermore, while Gaussian-Bernoulli RBMs (GBRBMs) exist for real-valued inputs, they are notoriously challenging to train effectively.

To address these limitations, this paper introduces the pcGRBM (Pairwise Constraints Restricted Boltzmann Machine with Gaussian Visible Units). This semi-supervised model incorporates background knowledge directly into the feature learning process using "must-link" and "cannot-link" constraints, ensuring that the learned representations are optimized for complex clustering tasks.

Methodology: The pcGRBM Framework

The core contribution of this work is a model that bridges the gap between unsupervised feature learning and supervised knowledge. The pcGRBM replaces standard binary visible units with linear units that incorporate Gaussian noise, making it suitable for high-dimensional image data.

The pcGRBM framework for feature extraction and clustering.

As illustrated in the framework above, the model's objective function is augmented with a penalty term based on pairwise constraints. This term encourages the model to minimize the distance between "must-link" pairs and maximize the distance between "cannot-link" pairs within the reconstructed feature space. The model is trained using a modified version of Contrastive Divergence (CD1), where gradient descent updates the weights ($W$) and biases ($a, b$) to satisfy these constraints while simultaneously reconstructing the input data. Once trained, the hidden layer activations serve as the input for standard clustering algorithms such as K-means, Spectral Clustering (SP), and Affinity Propagation (AP).

Experimental Results and Analysis

The authors evaluated the pcGRBM using twelve image datasets from the Microsoft Research Asia Multimedia (MSRA-MM) collection, featuring high-dimensional vectors (892 to 899 features) representing various image properties.

Accuracy comparison across 12 datasets under varying percentages of pairwise constraints.

The experimental results, shown in the plots above, demonstrate a consistent performance boost across all datasets. On average, K-means clustering using pcGRBM features achieved an accuracy of 50.03%, compared to 46.57% for standard RBM features. Beyond accuracy, cluster "purity" also saw significant gains; for instance, Spectral Clustering based on pcGRBM reached a purity of 0.8106, compared to 0.7709 for the baseline. Statistical significance was confirmed using the Friedman Aligned Ranks test, proving that these gains are robust.

Key Insights and Discussion

A primary takeaway from this research is the effectiveness of encoding knowledge into features. By using pairwise constraints to shape the feature space itself, similar items are encouraged to "flock together" in the hidden representation before any clustering algorithm is applied. This approach often outperformed dedicated semi-supervised clustering algorithms (like Cop-Kmeans) that use the same constraints, suggesting that the quality of the feature representation is a more critical factor than the specific clustering logic.

Furthermore, the model treats these constraints as "soft" targets within the energy function. This design choice allows the pcGRBM to remain robust even when constraints are noisy or the underlying data distribution is highly complex.

Conclusion and Future Work

The pcGRBM provides a robust method for guiding RBM feature learning with minimal supervision. While the current results are promising for single-layer architectures, future work could extend this approach to Deep Belief Networks (DBNs) or Deep Boltzmann Machines (DBMs) to further enhance feature expression. Additionally, exploring the impact of hidden layer dimensionality and comparing the model against modern deep semi-supervised embedding methods, such as Triplet Loss networks, remains a valuable direction for future research.

Summary updated

Abstract

Restricted Boltzmann machines (RBMs) and their variants are usually trained by contrastive divergence (CD) learning, but the training procedure is an unsupervised learning approach, without any guidances of the background knowledge. To enhance the expression ability of traditional RBMs, in this paper, we propose pairwise constraints (PCs) RBM with Gaussian visible units (pcGRBM) model, in which the learning procedure is guided by PCs and the process of encoding is conducted under these guidances. The PCs are encoded in hidden layer features of pcGRBM. Then, some pairwise hidden features of pcGRBM flock together and another part of them are separated by the guidances. In order to deal with real-valued data, the binary visible units are replaced by linear units with Gaussian noise in the pcGRBM model. In the learning process of pcGRBM, the PCs are iterated transitions between visible and hidden units during CD learning procedure. Then, the proposed model is inferred by approximative gradient descent method and the corresponding learning algorithm is designed. In order to compare the availability of pcGRBM and traditional RBMs with Gaussian visible units, the features of the pcGRBM and RBMs hidden layer are used as input “data” for ${K}$ -means, spectral clustering (SP) and affinity propagation (AP) algorithms, respectively. We also use tenfold cross-validation strategy to train and test pcGRBM model to obtain more meaningful results with PCs which are derived from incremental sampling procedures. A thorough experimental evaluation is performed with 12 image datasets of Microsoft Research Asia Multimedia. The experimental results show that the clustering performance of ${K}$ -means, SP, and AP algorithms based on pcGRBM model are significantly better than traditional RBMs. In addition, the pcGRBM model for clustering tasks shows better performance than some semi-supervised clustering algorithms.
BibTeX
@article{chu2019restricted,
  author = {Jielei Chu and Hongjun Wang and Hua Meng and Peng Jin and Tianrui Li},
  title = {{Restricted Boltzmann Machines With Gaussian Visible Units Guided by Pairwise Constraints}},
  journal = {IEEE Trans. Cybernetics},
  year = {2019},
  url = {https://openreview.net/forum?id=RXCgboO3S_},
}
Figure