Micro-Supervised Disturbance Learning: A Perspective of Representation Probability Distribution

Jielei Chu, Jing Liu, Hongjun Wang, Hua Meng, Zhiguo Gong, Tianrui Li

TL;DR

This paper introduces "Micro-Supervised Disturbance Learning" (Micro-DL), a novel approach to representation learning that leverages subtle "small perturbations" to improve model stability and performance, especially in low-label scenarios. By fine-tuning the probability distribution of learned representations, Micro-DL significantly outperforms existing methods in clustering tasks.

What Problem Does This Paper Address?

Traditional representation learning methods often struggle with instability, particularly when relying on Euclidean distance, and are hampered by the scarcity and high cost of labeled data. This makes it challenging to learn robust and expressive representations, especially for tasks like clustering where clear data separation is crucial. The authors aim to develop a more stable and label-efficient representation learning technique that can extract meaningful features even with limited supervision.

Methods

The core innovation is the introduction of "small-perturbation ideology" to representation learning, specifically focusing on the probability distribution of representations rather than just their direct values.

  • Micro-Supervised Disturbance Learning (Micro-DL): This framework uses "positive small-perturbation information" (SPI) derived from just two labels per cluster.
  • Micro-DGRBM and Micro-DRBM Models: Two variant Restricted Boltzmann Machine (RBM) models are proposed:
  • Micro-DGRBM: For real-valued data, using Gaussian visible units and binary hidden units.
  • Micro-DRBM: For binary data, using binary visible and hidden units.
  • Stimulating Representation Distributions: The SPI is used to:
  • Minimize KL Divergence: Within the same cluster, the Kullback-Leibler (KL) divergence between the probability distributions of perturbed representations is minimized, encouraging them to become more similar.
  • Maximize KL Divergence: Between different clusters, the KL divergence is maximized, pushing their representation distributions further apart.
  • Deep Micro-DL Framework: A deep architecture is built by stacking these shallow Micro-DL models (one Micro-DGRBM followed by multiple Micro-DRBMs) to learn representations hierarchically. This deep model is trained using Contrastive Divergence (CD) learning.

Datasets

The paper evaluates the proposed Micro-DL architecture on a variety of datasets:

  • MSRA-MM: A collection of image sub-datasets (aquarium, bathroom, blog, blood, bouquet, bugat, cactus, voituretuning).
  • UCI Datasets:
    • car (simple hierarchical decision mode)
    • KDD99 (network intrusion detection)
    • segmentation (image segmentation)
    • vowe (speaker-independent recognition)

Key Insights and Ideas

  • Perturbations as a Learning Signal: The key insight is that small, controlled perturbations to data, when analyzed through the lens of representation probability distributions, can act as a powerful learning signal, especially for improving stability and discriminative power.
  • Focus on Distribution, Not Just Values: Moving beyond Euclidean distance to analyze the distribution of representations (using KL divergence) allows for a more nuanced understanding of similarity and dissimilarity between data points.
  • Label Efficiency: The SPI requires only two labels per cluster, making the approach highly effective in low-label or semi-supervised settings.
  • Stability through Distributional Alignment: By encouraging similar distributions within clusters and dissimilar ones between clusters, the method inherently improves the stability of learned representations.

Key Results

The Micro-DL architecture consistently outperformed baseline methods across multiple clustering metrics:

  • Clustering Accuracy: Micro-DL achieved an average accuracy of 0.7343, a significant improvement over the next best method (Semi-MG at 0.6692) and a substantial leap from the initial Semi-SP baseline (0.3724).
  • Jaccard Index (Jac): Micro-DL reached 0.5836, outperforming Semi-MG (0.5073) and NMicro-DL (0.4015).
  • Fowlkes-Mallows (FM) Index: Micro-DL achieved 0.7564, surpassing Semi-MG (0.6807) and NMicro-DL (0.5665).
  • Rand Index: Micro-DL reached 0.5988, outperforming Semi-MG (0.5639) and NMicro-DL (0.5408).
  • Friedman Aligned Ranks Test: Micro-DL ranked first among all tested algorithms, indicating its superior overall performance across the datasets.
  • Comparison with NMicro-DL: The proposed Micro-DL consistently outperformed its non-stimulated counterpart (NMicro-DL), demonstrating the effectiveness of the SPI in enhancing representation learning.

Limitations and Future Work

  • Scale Coefficient Sensitivity: The performance of Micro-DL is sensitive to the scale coefficient ($\alpha$) that balances the CD loss and the SPI loss. The paper shows that $\alpha=0.3$ generally yields optimal results, but this parameter might require tuning for different datasets.
  • Future Work: The authors suggest several avenues for future research:
  • Applying the small-perturbation ideology to other neural network architectures.
  • Investigating Micro-supervised representation learning with only a single label per cluster.
  • Exploring the performance of Micro-DL on larger-scale datasets.

Summary updated

Abstract

The instability is shown in the existing methods of representation learning based on Euclidean distance under a broad set of conditions. Furthermore, the scarcity and high cost of labels prompt us to explore more expressive representation learning methods which depends on as few labels as possible. To address above issues, the small-perturbation ideology is firstly introduced on the representation learning model based on the representation probability distribution. The positive small-perturbation information (SPI) which only depend on two labels of each cluster is used to stimulate the representation probability distribution and then two variant models are proposed to fine-tune the expected representation distribution of Restricted Boltzmann Machine (RBM), namely, Micro-supervised Disturbance Gaussian-binary RBM (Micro-DGRBM) and Micro-supervised Disturbance RBM (Micro-DRBM) models. The Kullback-Leibler (KL) divergence of SPI is minimized in the same cluster to promote the representation probability distributions to become more similar in Contrastive Divergence (CD) learning. In contrast, the KL divergence of SPI is maximized in the different clusters to enforce the representation probability distributions to become more dissimilar in CD learning. To explore the representation learning capability under the continuous stimulation of the SPI, we present a deep Micro-supervised Disturbance Learning (Micro-DL) framework based on the Micro-DGRBM and Micro-DRBM models and compare it with a similar deep structure which has no external stimulation. Experimental results demonstrate that the proposed deep Micro-DL architecture shows better performance in comparison to the baseline method, the most related shallow models and deep frameworks for clustering.
BibTeX
@article{chu2023micro,
  author = {Jielei Chu and Jing Liu and Hongjun Wang and Hua Meng and Zhiguo Gong and Tianrui Li},
  title = {{Micro-Supervised Disturbance Learning: A Perspective of Representation Probability Distribution}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2022.3225461},
}

Loading related content...

Figure