Abstract
The instability is shown in the existing methods of representation learning based on Euclidean distance under a broad set of conditions. Furthermore, the scarcity and high cost of labels prompt us to explore more expressive representation learning methods which depends on the labels as few as possible. To address these issues, the small-perturbation ideology is firstly introduced on the representation learning model based on the representation probability distribution. The positive small-perturbation information (SPI) which only depend on two labels of each cluster is used to stimulate the representation probability distribution and then two variant models are proposed to fine-tune the expected representation distribution of RBM, namely, Micro-supervised Disturbance GRBM (Micro-DGRBM) and Micro-supervised Disturbance RBM (Micro-DRBM) models. The Kullback-Leibler (KL) divergence of SPI is minimized in the same cluster to promote the representation probability distributions to become more similar in Contrastive Divergence (CD) learning. In contrast, the KL divergence of SPI is maximized in the different clusters to enforce the representation probability distributions to become more dissimilar in CD learning. To explore the representation learning capability under the continuous stimulation of the SPI, we present a deep Micro-supervised Disturbance Learning (Micro-DL) framework based on the Micro-DGRBM and Micro-DRBM models and compare it with a similar deep structure which has not any external stimulation. Experimental results demonstrate that the proposed deep Micro-DL architecture shows better performance in comparison to the baseline method, the most related shallow models and deep frameworks for clustering.
Introduction
Representation learning—the process of discovering optimal data representations for specific tasks—is foundational to modern machine learning. However, two major hurdles persist: the instability of Euclidean distance in high-dimensional spaces and the extreme scarcity of labeled data. Deep learning models are notoriously data-hungry, yet obtaining high-quality labels is often expensive or practically impossible in specialized fields.
To address these issues, this research introduces the Micro-supervised Disturbance Learning (Micro-DL) framework. Borrowing from the "small-perturbation ideology" found in physical systems, the authors propose that a small external stimulus can help a learning system settle into a more desirable and stable state.
The Micro-DL Framework
The core innovation of Micro-DL is the use of Small-Perturbation Information (SPI) to guide the learning process. Unlike standard unsupervised models that rely solely on reconstruction error, Micro-DL incorporates Kullback-Leibler (KL) divergence as a supervisory stimulus using as few as two labels per cluster.
As shown in Figure 1, the framework consists of a deep stack of Restricted Boltzmann Machines (RBMs). The architecture begins with a Micro-DGRBM (Gaussian RBM) to handle real-valued input data, followed by a series of Micro-DRBMs (Binary RBMs) to extract abstract features. The SPI acts as a continuous stimulus throughout the stack:
- Intra-cluster Similarity: For samples in the same cluster, the model minimizes the KL divergence of their representation distributions, forcing them to become more similar.
- Inter-cluster Dissimilarity: For samples in different clusters, the model maximizes the KL divergence, pushing their representation distributions apart.
By focusing on the probability distribution of hidden representations rather than simple spatial coordinates, the model gains significant stability against noise and high dimensionality.
Experimental Results
The researchers evaluated Micro-DL across 12 diverse datasets, including image-based datasets from MSRA-MM and various UCI repository datasets such as KDD99 (network intrusion) and Vowe (speaker recognition). The framework was compared against unsupervised baselines (VGAE, NMicro-DL) and semi-supervised shallow models (Semi-SP, pcGRBM).
Key Findings
- Clustering Accuracy: Micro-DL achieved an average accuracy of 0.7343 across all datasets, nearly doubling the performance of the baseline Semi-SP (0.3724) and significantly outperforming the unsupervised NMicro-DL (0.5619).
- Metric Improvements: Across Jaccard (Jac), Fowlkes-Mallows (FM), and Rand indices, Micro-DL consistently took the top spot. For example, the average Jac index rose to 0.5836, a massive leap from the 0.2521 seen in earlier semi-supervised methods.
- Statistical Significance: Using the Friedman Aligned Ranks test, Micro-DL ranked 1st with a score of 9.4167, while the next best model (Semi-MG) trailed at 25.25.
Conclusion
Micro-supervised Disturbance Learning demonstrates that minimal supervision—just two labels per cluster—is sufficient to "seed" and organize the entire representation space of a deep network. By integrating KL-based disturbances directly into the Contrastive Divergence learning algorithm, the model achieves superior clustering performance and stability. Future research may extend this small-perturbation ideology to other neural architectures, such as CNNs or Transformers, and explore the limits of extreme micro-supervision with even fewer labels.
Summary updated