Prototypical Contrastive Learning of Unsupervised Representations

Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, Steven C. H. Hoi

Abstract

This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it implicitly encodes semantic structures of the data into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning.

Introduction: Beyond Instance Discrimination

Recent breakthroughs in self-supervised learning have been dominated by instance-wise contrastive learning methods like MoCo and SimCLR. These frameworks learn by treating every image as a unique class, teaching the model that an image and its augmented versions are the "same," while all other images in the dataset are "different."

While effective for learning low-level textures, this approach suffers from class collision. For example, two different photos of dogs are treated as negatives of each other, forcing the model to push their embeddings apart despite their shared semantic meaning. This prevents the model from capturing the global semantic structure of the dataset. Prototypical Contrastive Learning (PCL) addresses this by bridging the gap between instance-level discrimination and semantic clustering.

Methods

PCL formulates unsupervised learning as an Expectation-Maximization (EM) process to find the maximum-likelihood estimation of the network parameters. The framework alternates between two main steps:

  1. E-Step (Clustering): The model performs $k$-means clustering on the current image embeddings to identify "prototypes" (cluster centroids). To capture hierarchical information (e.g., distinguishing between a "poodle," a "dog," and an "animal"), PCL performs clustering multiple times with varying numbers of clusters ($K$).
  2. M-Step (Representation Learning): The model optimizes the network parameters by minimizing the ProtoNCE loss, a generalized version of the InfoNCE loss. Instead of solely comparing an image to other instances, ProtoNCE compares an image embedding to its assigned prototypes across different granularities.
The PCL framework follows an EM-based approach, alternating between clustering (E-step) and representation optimization via ProtoNCE (M-step).

Hierarchical Prototypes and ProtoNCE

A key innovation in PCL is the use of multiple cluster granularities. By assigning an image to multiple prototypes (e.g., a fine-grained cluster and a coarse-grained cluster), the model learns an embedding space that respects semantic hierarchies.

Furthermore, PCL introduces concentration estimation ($\phi$). This dynamic variable measures the density of a cluster. If a cluster is loose, the loss function pulls embeddings toward the prototype more aggressively; if it is tight, the focus shifts, preventing the "collapsing" problem where data points converge into a single uninformative cluster.

PCL bridges the gap between instance-wise discrimination and semantic clustering by utilizing hierarchical prototypes.

Key Insights

  • Prototypes as Latent Variables: By treating cluster assignments as latent variables, PCL provides a rigorous theoretical EM framework for unsupervised representation learning.
  • Semantic Information Gain: Mutual Information (MI) analysis demonstrates that prototypes contain significantly more information about ground-truth classes than raw instance features, explaining PCL's superior performance in semantic tasks.
  • Dynamic Concentration: The introduction of $\phi$ ensures balanced, semantically meaningful groupings by adjusting the "pull" of the prototype based on local cluster density.

Experimental Results

PCL demonstrates significant advantages over "instance-only" methods across several benchmarks:

  • Low-Resource Transfer Learning: On the VOC07 benchmark with only 1 labeled sample per class, PCL achieved 46.9% mAP, a massive improvement over MoCo's 31.4%.
  • Semi-Supervised Learning: When fine-tuning on ImageNet with only 1% of labels, PCL reached 75.3% top-5 accuracy, setting a new state-of-the-art at the time of publication.
  • Clustering Quality: PCL achieved an Adjusted Mutual Information (AMI) score of 0.410, significantly outperforming DeepCluster (0.281) and MoCo (0.285), indicating a much more organized embedding space.
  • Object Detection: PCL effectively closed the gap between self-supervised and supervised pre-training on VOC object detection, reaching 71.7% mAP (compared to 72.8% for supervised models).

Conclusion and Future Work

Prototypical Contrastive Learning successfully integrates clustering into the contrastive learning paradigm, allowing models to learn representations that are both discriminative and semantically structured. While the computational overhead of $k$-means on large datasets is a consideration, the use of efficient libraries like Faiss makes PCL a viable and powerful alternative to standard contrastive methods.

Future work may involve extending PCL to other domains such as Natural Language Processing (NLP) or exploring non-parametric clustering techniques to further refine the discovery of latent semantic structures.

Summary updated

Abstract

This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it implicitly encodes semantic structures of the data into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at https://github.com/salesforce/PCL.
BibTeX
@inproceedings{li2021prototypical,
  author = {Junnan Li and Pan Zhou and Caiming Xiong and Richard Socher and Steven C. H. Hoi},
  title = {{Prototypical Contrastive Learning of Unsupervised Representations}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=KmykpuSrjcq},
}
Figure