RPSC: Robust Pseudo-Labeling for Semantic Clustering

Sihang Liu, Wenming Cao, Ruigang Fu, Kaixiang Yang, Zhiwen Yu

Introduction

Deep clustering aims to learn feature representations and cluster assignments simultaneously without ground-truth labels. A common strategy is pseudo-labeling, where the model treats its own high-confidence predictions as ground truth for subsequent training iterations. However, this approach is susceptible to "error accumulation": if a model incorrectly labels an ambiguous sample (e.g., a husky mistaken for a wolf) and then trains on that label with high confidence, it reinforces its own mistakes in a "garbage in, garbage out" loop.

To address this, the Robust Pseudo-labeling for Semantic Clustering (RPSC) framework introduces a two-stage process designed to filter noise and weight samples based on semantic confidence, preventing the model from over-fitting to early, incorrect guesses.

The RPSC Framework

RPSC operates in two distinct stages: RPSC-Self for initial label generation and RPSC-Semi for iterative refinement.

Stage 1: RPSC-Self and Soft Contrastive Loss

In the first stage, the model utilizes a pre-trained backbone (such as MoCo-v2) to extract initial features. It then calculates cluster centers within the embedding space. The core innovation here is the Soft Contrastive Loss (SCL). Unlike standard contrastive losses that treat all positive pairs equally, SCL incorporates a "semantic confidence" score ($d_{ic}$).

Samples located near cluster centers are assigned high confidence and exert a stronger influence on the model's weight updates. Conversely, samples near cluster boundaries are down-weighted, preventing the model from over-fitting to ambiguous data. Furthermore, SCL intelligently excludes "intra-class" samples from the loss denominator to prevent the accidental repulsion of similar images. To ensure stability, the framework employs multi-head training, selecting the projection head that minimizes the SCL value.

The RPSC architecture: (a) RPSC-Self generates robust labels using Soft Contrastive Loss (SCL); (b) RPSC-Semi refines these labels through local consistency filtering and semi-supervised training.

Stage 2: RPSC-Semi and Local Consistency

The second stage refines the initial pseudo-labels by identifying a "gold standard" set of reliable samples. This is achieved through a local-consistency filtering mechanism: a pseudo-label is deemed reliable only if the sample's nearest neighbors in the embedding space share the same label. These reliable samples then guide a semi-supervised learning phase (using the FreeMatch framework), which adaptively adjusts classification thresholds for each category to further sharpen cluster boundaries.

Experimental Results

RPSC was evaluated across six benchmarks, including CIFAR-10, STL-10, and ImageNet subsets. The framework demonstrated significant improvements over 18 baseline algorithms, including SCAN and SPICE.

The most notable performance gain occurred on the ImageNet-Dogs dataset, a challenging benchmark requiring fine-grained discrimination between 15 dog breeds. RPSC achieved an accuracy of 0.688, representing a 24% improvement over previous state-of-the-art methods. On CIFAR-100 (using 20 super-classes), RPSC reached an accuracy of 0.556, outperforming the previous best by nearly 2%.

t-SNE visualizations showing the progression of semantic clustering from epoch 0 to 100, with Normalized Mutual Information (NMI) improving from 0.212 to 0.744.

Ablation studies confirmed that the SCL loss is a versatile component; integrating it into existing frameworks like SCAN and SPICE consistently yielded performance boosts, highlighting its effectiveness in managing boundary-sample noise.

Conclusion and Future Work

While RPSC sets new records on several benchmarks, it faces challenges with high-cardinality datasets like Tiny-ImageNet (200 clusters). In such complex scenarios, if the initial RPSC-Self stage fails to identify enough reliable "anchors," the subsequent refinement stage lacks the necessary foundation for improvement. Future research will explore scaling RPSC to handle larger cluster counts and extending the framework to multi-view clustering and transfer learning applications.

Summary updated

Abstract

Clustering methods achieve performance improvement by jointly learning representation and cluster assignment. However, they do not consider the confidence of pseudo-labels which are not optimal as supervised information, resulting into error accumulation. To address this issue, we propose a Robust Pseudo-labeling for Semantic Clustering (RPSC) approach, which includes two stages. In the first stage (RPSC-Self), we design a semantic pseudo-labeling scheme by using the consistency of samples, i.e., samples with same semantics should be close to each other in the embedding space. To exploit robust semantic pseudo-labels for self-supervised learning, we propose a soft contrastive loss (SCL) which encourage the model to believe high-confidence sematic pseudo-labels and be less driven by low-confidence pseudo-labels. In the second stage (RPSC-Semi), we first determine the semantic pseudo-label of a sample based on the distance between itself and cluster centers, followed by screening out reliable semantic pseudo-label by exploiting the consistency. These reliable pseudo-labels are used as supervised information in the pseudo-semi-supervised learning algorithm to further improve the performance. Experimental results show that RPSC outperforms 18 competitive clustering algorithms significantly on six challenging image benchmarks. In particular, RPSC achieves an accuracy of 0.688 on ImageNet-Dogs, which is an up to 24% improvement, compared with the second-best method. Meanwhile, we conduct ablation studies to investigate effects of different augmented strategies on RPSC as well as contributions of terms in SCL to clustering performance. Besides, experimental results indicate that SCL can be easily integrated into existing clustering methods and bring performance improvement.
BibTeX
@inproceedings{liu2024rpsc,
  author = {Sihang Liu and Wenming Cao and Ruigang Fu and Kaixiang Yang and Zhiwen Yu},
  title = {{RPSC: Robust Pseudo-Labeling for Semantic Clustering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i12.29309},
}
Figure