Abstract
Contrastive learning is one of the fastest growing research areas in machine learning due to its ability to learn useful representations without labeled data. However, contrastive learning is susceptible to shortcuts—i.e., it may learn shortcut features irrelevant to the task of interest and discard relevant information. Past work has addressed this limitation via handcrafted data augmentations that eliminate the shortcut. But manually crafted augmentations do not work across all datasets and tasks. Further, data augmentations fail in addressing shortcuts in multi-attribute classification when one attribute acts as a shortcut around other attributes. In this paper, we analyze the objective function of contrastive learning and formally prove that it is vulnerable to shortcuts. We then present reconstructive contrastive learning (RCL), a framework for learning unsupervised representations that are robust to shortcuts. The key idea is to force the learned representation to reconstruct the input, which naturally counters potential shortcuts. Extensive experiments verify that RCL is highly robust to shortcuts and outperforms state-of-the-art contrastive learning methods on a variety of datasets and tasks.
The Shortcut Problem in Contrastive Learning
Self-supervised contrastive learning (CL) has revolutionized computer vision by enabling models to learn powerful representations without human-annotated labels. The core principle involves teaching a model to recognize that different views (augmentations) of the same image should have similar embeddings, while views of different images should be distinct.
However, CL suffers from a fundamental "laziness" known as shortcuts. Because the standard contrastive loss (InfoNCE) encourages the model to find the minimum amount of information necessary to distinguish between images, the model often latches onto simple, irrelevant features—like background textures or color histograms—rather than deep semantic features.
Current solutions rely on handcrafted data augmentations (e.g., color jittering, cropping) to manually "break" these shortcuts. This approach has two major flaws:
- Domain Specificity: Augmentations optimized for natural images often fail on specialized domains like medical imaging or sensor data.
- Multi-attribute Interference: In datasets with multiple attributes (e.g., a face with age, gender, and ethnicity), one attribute may act as a shortcut for others. Augmentations cannot easily isolate and remove these semantic shortcuts without destroying the features we intend to learn.
Reconstructive Contrastive Learning (RCL)
To address these vulnerabilities, the authors propose Reconstructive Contrastive Learning (RCL). The core insight is that while contrastive learning is an "information-discarding" process, reconstruction is an "information-retaining" process. By combining them, RCL forces the model to maintain a comprehensive representation of the input, preventing it from ignoring complex semantic features in favor of easy shortcuts.
Architecture
The RCL framework employs a dual-branch architecture sharing a single encoder $E_\theta$:
- The Contrastive Branch: Follows standard CL practices (like SimCLR), pulling positive pairs together in the latent space.
- The Reconstructive Branch: Implements an inpainting task. Random patches of the input image are masked, and a decoder $D_\delta$ must reconstruct the missing pixels from the latent features.
Two-Phase Training Strategy
The authors found that joint training alone is insufficient to overcome the "saddle point" local minima created by shortcuts. They introduce a two-phase approach:
- Warm-up Phase: The model is trained solely on the reconstructive branch. This ensures the encoder captures a holistic representation of the image data.
- Joint Training Phase: Both branches are trained together. Since the encoder already possesses a rich representation, it is less likely to discard important features during the contrastive phase.
Experimental Evaluation
The researchers evaluated RCL against state-of-the-art methods on benchmarks including ImageNet, MPII, FairFace, and a synthetic "stress test" called Colorful-MNIST. Colorful-MNIST places MNIST digits on complex backgrounds from the STL-10 dataset to lure models into using the background as a shortcut.
Key Results
- Solving Colorful-MNIST: Standard CL methods fail this test, achieving only ~15% accuracy because they focus entirely on the background shortcut. In contrast, RCL achieves 88.3% accuracy, successfully capturing the underlying semantic digit.
- Robustness to Augmentation: On ImageNet, removing color distortion causes standard MoCo's accuracy to drop by over 10%. RCL's performance is significantly more stable, dropping by only ~5%, proving it relies less on handcrafted "tricks."
- Multi-Attribute Balance: On FairFace, RCL significantly outperformed baselines in Age and Gender classification because it prevented the "Ethnicity" attribute from acting as a dominant shortcut.
Conclusion
Reconstructive Contrastive Learning provides a robust alternative to purely contrastive methods by balancing information discarding with information retention. By integrating an inpainting task, RCL effectively "nudges" the model out of shortcut-driven local minima, leading to representations that are more generalizable and less dependent on handcrafted augmentation tricks. Future work will explore adaptive loss balancing and extending the framework to non-visual modalities.
Summary updated