Introduction
In self-supervised learning, Contrastive Learning has emerged as a dominant paradigm, typically focusing on "instance-wise" tasks that align different views of the same image while pushing distinct images apart. Prototypical Contrastive Learning (PCL) extends this by comparing images to "prototypes"—clusters representing broader semantic concepts. However, current PCL methods often suffer from prototype collapse, or "coagulation." This occurs when the model optimizes too aggressively, causing samples to cluster into a single point in the embedding space, thereby destroying internal category diversity and leaving large voids on the representation hypersphere.
The PAUC Framework
To address the coagulation problem, we introduce PAUC (Prototypical representation through Alignment, Uniformity, and Correlation). This framework modifies the standard ProtoNCE loss with three critical constraints designed to ensure prototypes are well-distributed and semantically rich:
- Prototypical Alignment: Pulls together embeddings from similar prototypes to maintain a consistent semantic core.
- Prototypical Uniformity: Uses physics-inspired kernels to push prototypes apart, ensuring they occupy the embedding hypersphere evenly and preventing "bunching."
- Prototypical Correlation: Minimizes correlation between feature vector dimensions, forcing the model to learn diverse, non-redundant features for each prototype.
Additionally, we introduce the Normalized Earth Mover’s Distance (NEMD) as a diagnostic metric to quantify the degree of representation collapse by treating prototypes as probability distributions.
Experimental Results
PAUC was evaluated on standard computer vision benchmarks using a ResNet-50 backbone. The results demonstrate significant improvements over both instance-wise and prototypical state-of-the-art (SOTA) methods:
- ImageNet-100: PAUC achieved a Top-1 accuracy of 84.46%, outperforming PCL-v2 by over 6%.
- ImageNet-1K: On the full benchmark, PAUC reached 75.16% Top-1 accuracy, a 2.46% improvement over SwAV and a 13.66% jump over the baseline PCL.
As shown in the visualization above, standard embedding spaces often lead to "clumpy" or collapsed distributions. PAUC successfully creates more distinct, evenly spaced clusters that better reflect the underlying data space while maintaining discriminative features.
Conclusion
By rethinking PCL through the lenses of alignment, uniformity, and correlation, PAUC effectively mitigates the coagulation trap. While PCL methods involve additional clustering overhead, the substantial gains in representation quality and linear classification accuracy on ImageNet benchmarks suggest that enforcing hypersphere efficiency is vital for the next generation of self-supervised models.
Summary updated