Abstract
We propose a novel framework for image clustering that incorporates joint representation learning and clustering. Our method consists of two heads that share the same backbone network - a "representation learning" head and a "clustering" head. The "representation learning" head captures fine-grained patterns of objects at the instance level which serve as clues for the "clustering" head to extract coarse-grain information that separates objects into clusters. The whole model is trained in an end-to-end manner by minimizing the weighted sum of two sample-oriented contrastive losses applied to the outputs of the two heads. To ensure that the contrastive loss corresponding to the "clustering" head is optimal, we introduce a novel critic function called "log-of-dot-product". Extensive experimental results demonstrate that our method significantly outperforms state-of-the-art single-stage clustering methods across a variety of image datasets, improving over the best baseline by about 5-7% in accuracy on CIFAR10/20, STL10, and ImageNet-Dogs.
Introduction
Image clustering—grouping unlabeled images into semantic categories—is a fundamental challenge in unsupervised learning. While deep learning has improved this field, existing methods often fall into two traps: the over-retention of noise (e.g., background textures) and low intra-cluster similarity. The proposed Contrastive Representation Learning and Clustering (CRLC) framework addresses these by forcing a model to look at images through two lenses simultaneously: instance-level identity and cluster-level identity.
The CRLC Framework
The core of the CRLC framework is a shared backbone network (such as ResNet) that branches into two specialized heads.
- Representation Learning (RL-head): This head outputs a continuous feature vector focusing on instance-level patterns and fine-grained details.
- Clustering (C-head): This head outputs a probability vector representing cluster assignments, capturing coarse-grained categories.
The backbone acts as a bridge, allowing the RL-head to transfer discriminative features (like specific object parts) to the C-head, enhancing its clustering decisions.
The "Log-of-Dot-Product" Critic
Standard contrastive learning often uses a "dot-product" critic. However, for probability vectors, the authors introduce a "log-of-dot-product" critic: $$f(\tilde{q}, q_i) = \log(\tilde{q}^\top q_i)$$ This function provides a tighter bound on Mutual Information for categorical distributions, preventing degenerate solutions where the model collapses all images into a single cluster. Additionally, gradient smoothing is applied to prevent the model from saturating once it becomes confident.
Experimental Results
CRLC was evaluated on standard benchmarks including CIFAR-10/20, STL-10, and ImageNet subsets.
The method set new performance bars for single-stage clustering:
- Accuracy Gains: On CIFAR-10, CRLC achieved 79.9% accuracy, a significant improvement over the previous state-of-the-art (72.7%).
- Large Scale Success: In a two-stage configuration, it outperformed the SCAN method on ImageNet-200 (57.9% vs 56.3%).
- Semi-Supervised Efficiency: With only 4 labels per class on CIFAR-10, CRLC-semi achieved 10-30% lower error rates than established methods like MixMatch.
Discussion and Limitations
A key insight is the shift from cluster-oriented to sample-oriented comparisons in probability space, allowing the model to leverage more negative pairs. However, the model is sensitive to the weighting between the two heads and can occasionally make "collective errors," misclassifying groups of visually similar objects (e.g., confusing horses for dogs). Future work will explore explicit handling of neighboring samples and expansion into video and object detection domains.
Summary updated