Abstract
The instability is shown in the existing methods of representation learning based on Euclidean distance under a broad set of conditions. Furthermore, the scarcity and high cost of labels prompt us to explore more expressive representation learning methods which depends on the labels as few as possible. To address these issues, the small-perturbation ideology is firstly introduced on the representation learning model based on the representation probability distribution. The positive small-perturbation information (SPI) which only depend on two labels of each cluster is used to stimulate the representation probability distribution and then two variant models are proposed to fine-tune the expected representation distribution of RBM, namely, Micro-supervised Disturbance GRBM (Micro-DGRBM) and Micro-supervised Disturbance RBM (Micro-DRBM) models. The Kullback-Leibler (KL) divergence of SPI is minimized in the same cluster to promote the representation probability distributions to become more similar in Contrastive Divergence(CD) learning. In contrast, the KL divergence of SPI is maximized in the different clusters to enforce the representation probability distributions to become more dissimilar in CD learning. To explore the representation learning capability under the continuous stimulation of the SPI, we present a deep Micro-supervised Disturbance Learning (Micro-DL) framework based on the Micro-DGRBM and Micro-DRBM models and compare it with a similar deep structure which has not any external stimulation. Experimental results demonstrate that the proposed deep Micro-DL architecture shows better performance in comparison to the baseline method, the most related shallow models and deep frameworks for clustering.
Introduction
In the field of deep learning, representation learning—the process of teaching a machine to automatically discover the best features of data—is a cornerstone of success. However, two major hurdles remain: instability and label scarcity. Many traditional methods rely on Euclidean distance to measure similarity, which can be unstable under various data conditions. Furthermore, deep learning is notoriously "data-hungry," while high-quality labels are scarce and expensive to produce.
This research introduces a "micro-supervised" learning framework that addresses these issues by asking: Can we guide a model to learn better representations using the absolute minimum amount of human supervision? By treating a handful of labels as "small perturbations" that guide the probability distributions of data representations, the proposed model achieves state-of-the-art results without the need for massive datasets.
Methodology: Micro-supervised Disturbance Learning (Micro-DL)
The core idea of Micro-DL is inspired by physical systems, where a "small disturbance" can nudge a system into a new state. Instead of traditional geometric distance, the model focuses on the representation probability distribution.
Small-Perturbation Information (SPI)
Instead of labeling a significant percentage of a dataset, the authors use Small-Perturbation Information (SPI), which requires only two labeled examples per cluster. This minimal supervision acts as a "nudge" during training. During Contrastive Divergence (CD) learning, the model uses Kullback-Leibler (KL) divergence to measure the similarity between distributions. It minimizes KL divergence for pairs in the same cluster (making their "fingerprints" more similar) and maximizes it for pairs in different clusters (pushing them apart).
Hybrid Architecture
The framework utilizes two new variants of Restricted Boltzmann Machines (RBMs):
- Micro-DGRBM: Designed for real-valued, continuous data in the visible layer.
- Micro-DRBM: Designed for binary data in subsequent hidden layers.
As shown in the architecture above, these layers are stacked to create a deep Micro-DL framework. This allows the "disturbance" from the SPI to refine features across multiple levels of abstraction continuously through the network, rather than just at the final output layer.
Experimental Evaluation
The researchers evaluated the Micro-DL framework on 12 different datasets to ensure robustness, including eight sub-datasets from MSRA-MM (e.g., Aquarium, Blog, Blood) and four classic UCI benchmarks (Car Evaluation, KDD99, Image Segmentation, and Vowe).
Key Results
The Micro-DL framework consistently outperformed both traditional shallow models and modern deep unsupervised/semi-supervised frameworks:
- Accuracy Boost: Micro-DL achieved an average clustering accuracy of 0.7343, significantly higher than the baseline (Semi-SP) at 0.3742 and the unsupervised version (NMicro-DL) at 0.5619.
- Statistical Significance: Using the Friedman Aligned Ranks test, the authors demonstrated that Micro-DL ranks first among seven competing algorithms across all 12 datasets.
- Efficiency: The model achieved these gains while drastically reducing reliance on labels, proving that a "micro" amount of supervision is incredibly potent when applied to underlying probability distributions.
Conclusion and Future Work
Micro-supervised Disturbance Learning demonstrates that focusing on distributional stability rather than Euclidean coordinates leads to more robust representation learning. By applying SPI continuously through deep layers, the model creates highly discriminative features with minimal human intervention.
Future research will explore whether single-label supervision per cluster is feasible and investigate methods to automatically tune hyperparameters like the scale coefficient ($\alpha$). Additionally, the authors plan to adapt the small-perturbation ideology to other neural architectures, such as Convolutional Neural Networks (CNNs), for larger-scale big data applications.
Summary updated