Selective-Supervised Contrastive Learning with Noisy Labels

Shikun Li, Xiaobo Xia, Shiming Ge, Tongliang Liu

Introduction

Deep learning models are notoriously data-hungry, requiring massive datasets with high-quality annotations to achieve peak performance. While clean labels are expensive and time-consuming to produce, noisy labels—often derived from web scraping or user tags—are abundant but problematic. In standard supervised learning, these incorrect signals corrupt the model's internal representation of the data, leading to poor generalization.

This challenge is particularly acute for Supervised Contrastive Learning (Sup-CL). Sup-CL teaches models to pull representations of the same class together and push different classes apart. Because it operates on pairs, even a small percentage of noise can create a massive number of misleading relationships, forcing the model to learn incorrect semantic maps.

The Sel-CL Framework

To address this, the authors propose Selective-Supervised Contrastive Learning (Sel-CL). Sel-CL is designed to be robust to noisy labels by identifying "confident pairs" for training without requiring prior knowledge of the dataset's noise rate. The framework operates through a multi-stage process:

  1. Warm-up: The model begins with unsupervised contrastive learning to establish a baseline understanding of the data structure independent of labels.
  2. Selecting Confident Examples ($\mathcal{T}$): Using a k-Nearest Neighbors (kNN) approach, the model examines an image's neighbors in the representation space. If the neighbors' labels align with the image's given label, it is marked as a "confident example."
  3. Selecting Confident Pairs ($\mathcal{G}$): The model builds a set of training pairs from both confident examples sharing the same label and similarity-based pairs. The latter allows the model to recover useful information from noisy data by grouping images with extremely similar representations, even if their labels are suspect.
  4. Multi-Objective Learning: Training combines supervised contrastive loss on selected pairs, classification loss on confident examples, and a similarity loss to align predictions with the representation space.
The Sel-CL framework illustrates the pipeline from noisy inputs through representation learning to the selection of confident examples and pairs.

Key Insights and the "Positive Cycle"

A central contribution of Sel-CL is the demonstration of a "positive cycle" between representation learning and label selection. As the model learns better features, it becomes more adept at identifying incorrect labels; as these noisy labels are filtered out, the features become even cleaner. Unlike many existing methods, Sel-CL focuses on the relationship between samples (pairs) rather than just individual points, which proves more effective for contrastive frameworks.

Furthermore, Sel-CL is exceptionally robust to asymmetric noise (where labels are flipped to similar classes, like bird $\rightarrow$ airplane). By prioritizing representation similarity, the model can recognize that two images belong together regardless of their incorrect textual labels.

Performance and Results

Sel-CL consistently outperforms state-of-the-art methods like DivideMix and ELR across various benchmarks. On CIFAR-100 with 80% symmetric noise, Sel-CL achieved a weighted kNN accuracy of 62.49%, significantly higher than previous benchmarks. In real-world testing on the WebVision dataset, it reached a top-1 accuracy of 79.96%.

Precision curves show that the selection mechanism maintains extremely high accuracy for both examples and positive pairs throughout the training epochs.

Empirical evidence shows that the selection precision for both examples and pairs remains above 98% throughout training, validating the effectiveness of the denoising mechanism. Visualizations via t-SNE further confirm that Sel-CL creates tighter, more distinct class clusters compared to standard training methods under noisy conditions.

Limitations and Future Directions

While highly effective, Sel-CL does face computational hurdles. The kNN algorithm used for selection can be expensive on massive datasets, though fast kNN approximations offer a potential mitigation. Additionally, like most contrastive methods, it requires substantial GPU memory to support large batch sizes. Future work aims to extend this selective-supervised logic to other domains, including object detection and natural language processing.

Summary updated

Abstract

Deep networks have strong capacities of embedding data into latent representations and finishing following tasks. However, the capacities largely come from high-quality annotated labels, which are expensive to collect. Noisy labels are more affordable, but result in corrupted representations, leading to poor generalization performance. To learn robust representations and handle noisy labels, we propose selective-supervised contrastive learning (Sel-CL) in this paper. Specifically, Sel-CL extend supervised contrastive learning (Sup-CL), which is powerful in representation learning, but is degraded when there are noisy labels. Sel-CL tackles the direct cause of the problem of Sup-CL. That is, as Sup-CL works in a pair-wise manner, noisy pairs built by noisy labels mislead representation learning. To alleviate the issue, we select confident pairs out of noisy ones for Sup-CL without knowing noise rates. In the selection process, by measuring the agreement between learned representations and given labels, we first identify confident examples that are exploited to build confident pairs. Then, the representation similarity distribution in the built confident pairs is exploited to identify more confident pairs out of noisy pairs. All obtained confident pairs are finally used for Sup-CL to enhance representations. Experiments on multiple noisy datasets demonstrate the robustness of the learned representations by our method, following the state-of-the-art performance. Source codes are available at https://github.com/ShikunLi/Sel-Cl.
BibTeX
@misc{li2022selective,
  author = {Shikun Li and Xiaobo Xia and Shiming Ge and Tongliang Liu},
  title = {{Selective-Supervised Contrastive Learning with Noisy Labels}},
  year = {2022},
  eprint = {2203.04181},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.04181},
}
Figure