Stable Cluster Discrimination for Deep Clustering

Qi Qian

Abstract

Deep clustering can optimize representations of instances and explore the inherent data distribution simultaneously, demonstrating superior performance over conventional methods. However, the coupled objective often leads to a trivial solution where all instances collapse to uniform features. To tackle this, we propose a novel stable cluster discrimination (SeCu) task. We show that prevalent discrimination tasks are unstable for one-stage clustering due to the lack of ground-truth labels in mini-batches. SeCu mitigates this with a hardness-aware clustering criterion and a global entropy constraint. Extensive experiments on ImageNet and other benchmarks show that SeCu achieves state-of-the-art performance, validating the effectiveness of one-stage deep clustering.

Introduction

Deep clustering aims to learn feature representations and cluster assignments in a single, unified framework. While promising, one-stage methods often suffer from feature collapse, where the model maps all inputs to a single point to minimize loss trivially. Traditional solutions involve complex two-stage pipelines—pre-training followed by fine-tuning—which are computationally expensive and may have misaligned objectives.

The core instability in one-stage clustering stems from the "negative gradient trap." In unsupervised settings, we rely on pseudo-labels. In a typical training mini-batch, many clusters may have no positive instances. Standard cross-entropy loss continues to push these "empty" clusters away from the current batch's instances (negatives). Without positive anchors, this constant repulsion introduces significant gradient noise, destabilizing the learning process.

Methodology: Stable Cluster Discrimination (SeCu)

To address these instabilities, the SeCu framework introduces three key components:

  1. Stable Cluster Discrimination Loss: SeCu modifies the standard cross-entropy objective by applying a stop-gradient to negative instances during cluster center updates. This ensures that a cluster center is only influenced by instances assigned to it, preventing the "pushing" noise from irrelevant data.
  2. Hardness-Aware Criterion: Unlike standard k-means, which treats all points in a cluster equally, SeCu's loss function naturally prioritizes "hard" instances—those near cluster boundaries. This leads to more robust and discriminative boundaries.
  3. Global Entropy Regularization: To prevent cluster collapse (where all data points are assigned to one cluster), SeCu maximizes the global entropy of cluster assignments. This encourages a balanced distribution of data across all available clusters without the need for complex constraints like Sinkhorn-Knopp.
Comparison of SeCu against baseline k-means, showing improved accuracy and a more refined decision boundary.

Experimental Evaluation

The effectiveness of SeCu was evaluated on several standard computer vision benchmarks, including CIFAR-10, CIFAR-100, STL-10, and the large-scale ImageNet dataset.

Sample images from the benchmark datasets (CIFAR, ImageNet) used for evaluating the SeCu framework.

Key Results

SeCu demonstrates significant improvements over existing state-of-the-art (SOTA) methods:

  • ImageNet: SeCu achieved a top-1 clustering accuracy of 53.5%, outperforming the multi-stage SCAN method by 11%.
  • CIFAR-10: The model reached 93.0% accuracy, approaching the performance of fully supervised models.
  • Efficiency: As a one-stage method, SeCu is simpler to implement and faster to train than multi-stage alternatives, requiring no separate pre-training phase.

Conclusion

SeCu provides a robust solution for one-stage deep clustering by stabilizing the discrimination task. By addressing the negative gradient trap and incorporating hardness-aware updates with global entropy constraints, it achieves SOTA results on large-scale benchmarks. Future work will explore the application of SeCu to Vision Transformer (ViT) architectures and the integration of explicit nearest-neighbor information to further enhance clustering performance.

Summary updated

Abstract

Deep clustering can optimize representations of instances (i.e., representation learning) and explore the inherent data distribution (i.e., clustering) simultaneously, which demonstrates a superior performance over conventional clustering methods with given features. However, the coupled objective implies a trivial solution that all instances collapse to the uniform features. To tackle the challenge, a two-stage training strategy is developed for decoupling, where it introduces an additional pre-training stage for representation learning and then fine-tunes the obtained model for clustering. Meanwhile, one-stage methods are developed mainly for representation learning rather than clustering, where various constraints for cluster assignments are designed to avoid collapsing explicitly. Despite the success of these methods, an appropriate learning objective tailored for deep clustering has not been investigated sufficiently. In this work, we first show that the prevalent discrimination task in supervised learning is unstable for one-stage clustering due to the lack of ground-truth labels and positive instances for certain clusters in each mini-batch. To mitigate the issue, a novel stable cluster discrimination (SeCu) task is proposed and a new hardness-aware clustering criterion can be obtained accordingly. Moreover, a global entropy constraint for cluster assignments is studied with efficient optimization. Extensive experiments are conducted on benchmark data sets and ImageNet. SeCu achieves state-of-the-art performance on all of them, which demonstrates the effectiveness of one-stage deep clustering.
BibTeX
@inproceedings{qian2023stable,
  author = {Qi Qian},
  title = {{Stable Cluster Discrimination for Deep Clustering}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=WdWvPQlIvL},
}
Figure