Abstract
Deep clustering can optimize representations of instances and explore the inherent data distribution simultaneously, demonstrating superior performance over conventional methods. However, the coupled objective often leads to a trivial solution where all instances collapse to uniform features. To tackle this, we propose a novel stable cluster discrimination (SeCu) task. We show that prevalent discrimination tasks are unstable for one-stage clustering due to the lack of ground-truth labels in mini-batches. SeCu mitigates this with a hardness-aware clustering criterion and a global entropy constraint. Extensive experiments on ImageNet and other benchmarks show that SeCu achieves state-of-the-art performance, validating the effectiveness of one-stage deep clustering.
Introduction
Deep clustering aims to learn feature representations and cluster assignments in a single, unified framework. While promising, one-stage methods often suffer from feature collapse, where the model maps all inputs to a single point to minimize loss trivially. Traditional solutions involve complex two-stage pipelines—pre-training followed by fine-tuning—which are computationally expensive and may have misaligned objectives.
The core instability in one-stage clustering stems from the "negative gradient trap." In unsupervised settings, we rely on pseudo-labels. In a typical training mini-batch, many clusters may have no positive instances. Standard cross-entropy loss continues to push these "empty" clusters away from the current batch's instances (negatives). Without positive anchors, this constant repulsion introduces significant gradient noise, destabilizing the learning process.
Methodology: Stable Cluster Discrimination (SeCu)
To address these instabilities, the SeCu framework introduces three key components:
- Stable Cluster Discrimination Loss: SeCu modifies the standard cross-entropy objective by applying a stop-gradient to negative instances during cluster center updates. This ensures that a cluster center is only influenced by instances assigned to it, preventing the "pushing" noise from irrelevant data.
- Hardness-Aware Criterion: Unlike standard k-means, which treats all points in a cluster equally, SeCu's loss function naturally prioritizes "hard" instances—those near cluster boundaries. This leads to more robust and discriminative boundaries.
- Global Entropy Regularization: To prevent cluster collapse (where all data points are assigned to one cluster), SeCu maximizes the global entropy of cluster assignments. This encourages a balanced distribution of data across all available clusters without the need for complex constraints like Sinkhorn-Knopp.
Experimental Evaluation
The effectiveness of SeCu was evaluated on several standard computer vision benchmarks, including CIFAR-10, CIFAR-100, STL-10, and the large-scale ImageNet dataset.
Key Results
SeCu demonstrates significant improvements over existing state-of-the-art (SOTA) methods:
- ImageNet: SeCu achieved a top-1 clustering accuracy of 53.5%, outperforming the multi-stage SCAN method by 11%.
- CIFAR-10: The model reached 93.0% accuracy, approaching the performance of fully supervised models.
- Efficiency: As a one-stage method, SeCu is simpler to implement and faster to train than multi-stage alternatives, requiring no separate pre-training phase.
Conclusion
SeCu provides a robust solution for one-stage deep clustering by stabilizing the discrimination task. By addressing the negative gradient trap and incorporating hardness-aware updates with global entropy constraints, it achieves SOTA results on large-scale benchmarks. Future work will explore the application of SeCu to Vision Transformer (ViT) architectures and the integration of explicit nearest-neighbor information to further enhance clustering performance.
Summary updated