CLC: Cluster Assignment via Contrastive Representation Learning

Fei Ding, Dan Zhang, Yin Yang, Venkat Krovi, Feng Luo

Introduction

Clustering high-dimensional image data without ground-truth labels remains a fundamental challenge in computer vision. While self-supervised learning has made significant strides, scaling these methods to massive datasets like ImageNet often results in semantic drift or cluster collapse. The proposed framework, Contrastive Representation Disentanglement for Clustering (CDC), addresses these issues by explicitly separating categorical information from instance-specific features during the learning process.

The Challenge of Large-Scale Clustering

Traditional unsupervised clustering typically involves a two-stage pipeline: learning representations via self-supervised tasks and then applying a clustering algorithm like K-means. This approach suffers from two primary drawbacks:

  1. Semantic Drift: Learned features may over-index on low-level visual cues like color or texture rather than high-level semantics.
  2. The Hard Negative Problem: Standard contrastive learning treats all non-augmented pairs as negatives. This forces the model to push apart images that belong to the same semantic category (e.g., two different dogs), hindering the formation of cohesive clusters.

Methodology: Contrastive Representation Disentanglement

The core innovation of this work is the disentanglement of the feature space into two distinct components:

  • Categorical Representation ($z^c$): Encodes cluster assignments as logits.
  • Instance-specific Representation ($z^n$): Captures unique, fine-grained details of an individual image.
Figure 1: The architecture disentangles representations into categorical ($z^c$) and instance-specific ($z^n$) components using a dual-encoder setup to optimize a contrastive loss.

To optimize this disentangled space, the framework employs three key strategies:

  1. Contrastive Proxy Task: The model is trained to identify augmented versions of the same image using the combined information from $z^c$ and $z^n$.
  2. Equipartition Constraint: To avoid the trivial solution where all images are assigned to a single cluster (collapse), the Sinkhorn-Knopp algorithm is used to enforce an even distribution of samples across clusters within each training batch.
  3. Self-Adjusting Weights: The categorical component $z^c$ naturally acts as a weighting mechanism during training. It prioritizes "hard negatives"—images that are semantically similar but distinct instances—thereby sharpening the boundaries between different categories.

Experimental Results

The model was evaluated on several benchmarks, including CIFAR-10, STL-10, and the full ImageNet dataset.

Figure 2: Training progress on the full ImageNet-1K dataset, showing steady improvement in Normalized Mutual Information (NMI), Accuracy (ACC), and Adjusted Rand Index (ARI).

Performance on ImageNet

On the full ImageNet-1K dataset, the method achieved a state-of-the-art 53.4% accuracy, representing a +10.2% improvement over previous leading methods like SCAN. The training curves demonstrate robust convergence across multiple clustering metrics, including NMI and ARI.

Transferability and Efficiency

Beyond clustering, the features learned through this disentanglement process proved highly effective for downstream tasks. When used as a backbone for object detection on COCO and Pascal VOC, the model outperformed standard MoCo v2 baselines. Qualitatively, t-SNE visualizations confirm that the categorical space $z^c$ produces significantly cleaner and more semantically consistent groupings compared to traditional contrastive methods.

Conclusion

By integrating cluster assignment directly into the contrastive learning objective, this approach bridges the gap between representation learning and semantic grouping. The explicit disentanglement of categorical and instance-level features allows the model to handle the complexity of 1,000-class datasets like ImageNet with unprecedented accuracy. Future work may explore the integration of Vision Transformers (ViTs) and over-clustering strategies to further refine these semantic boundaries.

Summary updated

BibTeX
@misc{ding2023clc,
  author = {Fei Ding and Dan Zhang and Yin Yang and Venkat Krovi and Feng Luo},
  title = {{CLC: Cluster Assignment via Contrastive Representation Learning}},
  year = {2023},
  eprint = {2306.05439},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.05439},
}
Figure