Introduction
Deep learning models are notoriously data-hungry, requiring massive datasets with high-quality annotations to achieve peak performance. While clean labels are expensive and time-consuming to produce, noisy labels—often derived from web scraping or user tags—are abundant but problematic. In standard supervised learning, these incorrect signals corrupt the model's internal representation of the data, leading to poor generalization.
This challenge is particularly acute for Supervised Contrastive Learning (Sup-CL). Sup-CL teaches models to pull representations of the same class together and push different classes apart. Because it operates on pairs, even a small percentage of noise can create a massive number of misleading relationships, forcing the model to learn incorrect semantic maps.
The Sel-CL Framework
To address this, the authors propose Selective-Supervised Contrastive Learning (Sel-CL). Sel-CL is designed to be robust to noisy labels by identifying "confident pairs" for training without requiring prior knowledge of the dataset's noise rate. The framework operates through a multi-stage process:
- Warm-up: The model begins with unsupervised contrastive learning to establish a baseline understanding of the data structure independent of labels.
- Selecting Confident Examples ($\mathcal{T}$): Using a k-Nearest Neighbors (kNN) approach, the model examines an image's neighbors in the representation space. If the neighbors' labels align with the image's given label, it is marked as a "confident example."
- Selecting Confident Pairs ($\mathcal{G}$): The model builds a set of training pairs from both confident examples sharing the same label and similarity-based pairs. The latter allows the model to recover useful information from noisy data by grouping images with extremely similar representations, even if their labels are suspect.
- Multi-Objective Learning: Training combines supervised contrastive loss on selected pairs, classification loss on confident examples, and a similarity loss to align predictions with the representation space.
Key Insights and the "Positive Cycle"
A central contribution of Sel-CL is the demonstration of a "positive cycle" between representation learning and label selection. As the model learns better features, it becomes more adept at identifying incorrect labels; as these noisy labels are filtered out, the features become even cleaner. Unlike many existing methods, Sel-CL focuses on the relationship between samples (pairs) rather than just individual points, which proves more effective for contrastive frameworks.
Furthermore, Sel-CL is exceptionally robust to asymmetric noise (where labels are flipped to similar classes, like bird $\rightarrow$ airplane). By prioritizing representation similarity, the model can recognize that two images belong together regardless of their incorrect textual labels.
Performance and Results
Sel-CL consistently outperforms state-of-the-art methods like DivideMix and ELR across various benchmarks. On CIFAR-100 with 80% symmetric noise, Sel-CL achieved a weighted kNN accuracy of 62.49%, significantly higher than previous benchmarks. In real-world testing on the WebVision dataset, it reached a top-1 accuracy of 79.96%.
Empirical evidence shows that the selection precision for both examples and pairs remains above 98% throughout training, validating the effectiveness of the denoising mechanism. Visualizations via t-SNE further confirm that Sel-CL creates tighter, more distinct class clusters compared to standard training methods under noisy conditions.
Limitations and Future Directions
While highly effective, Sel-CL does face computational hurdles. The kNN algorithm used for selection can be expensive on massive datasets, though fast kNN approximations offer a potential mitigation. Additionally, like most contrastive methods, it requires substantial GPU memory to support large batch sizes. Future work aims to extend this selective-supervised logic to other domains, including object detection and natural language processing.
Summary updated