Contrastive Learning Improves Model Robustness Under Label Noise

Aritra Ghosh, Andrew S. Lan

Abstract

Training deep learning models on datasets with noisy labels typically leads to poor performance due to the memorization of errors. This paper demonstrates that pre-training models using self-supervised contrastive learning (SimCLR) provides a robust foundation for feature representation, enabling standard supervised methods to achieve significant performance gains. In high-noise scenarios, this approach outperforms complex state-of-the-art algorithms by substantial margins, highlighting the importance of decoupling feature learning from label handling.

Introduction

Deep neural networks are highly sensitive to label noise. When trained with standard Categorical Cross-Entropy (CCE) loss, they tend to memorize incorrect labels, which degrades generalization. Existing solutions generally fall into two categories: supervised robust methods (e.g., specialized loss functions like $L_q$) and complex semi-supervised learning (SSL) pipelines that treat noisy samples as unlabeled. While SSL methods currently lead in performance, they are often difficult to implement. This research investigates whether the failure of simpler supervised methods is due to the noise itself or a failure to learn quality visual representations when labels are unreliable.

Methodology

The authors propose a two-stage pipeline to decouple representation learning from label noise:

  1. Self-Supervised Pre-training: Using the SimCLR framework, the model learns to recognize augmented versions of the same image without using any labels. This ensures that the initial feature learning is not corrupted by label noise.
  2. Fine-tuning with Robust Methods: The pre-trained model is fine-tuned using supervised methods, including standard CCE, $L_q$ loss, and Meta-Weight Net (MWNet).

Experimental Results

The approach was evaluated on CIFAR-10, CIFAR-100, and the real-world Clothing1M dataset.

Performance in Extreme Noise

The most significant gains were observed in high-noise environments. On CIFAR-100 with 90% symmetric noise, standard CCE with random initialization achieves only 10.1% accuracy. However, initializing with SimCLR pre-training boosts this to 52.11%, a significant relative gain over the previous state-of-the-art, DivideMix (31.5%). Similarly, on CIFAR-10 with 90% noise, SimCLR pre-training improves CCE performance to 82.9%, surpassing DivideMix's 76%.

Performance comparison on CIFAR-10 and CIFAR-100 under symmetric noise.

Real-World Application

On the Clothing1M dataset, which contains naturally occurring noise from web-scraped data, the SimCLR-initialized models showed strong performance, demonstrating the practical utility of this strategy beyond synthetic benchmarks.

Accuracy on the real-world Clothing1M dataset.

Key Insights

  • The Representation Bottleneck: Supervised robust methods struggle primarily because they fail to build a solid feature foundation on noisy data. Contrastive pre-training provides this foundation, making even "fragile" losses like CCE resilient.
  • Decoupling is Key: Separating the learning of visual features from the learning of class labels prevents the model from memorizing noise during critical early training stages.
  • Simplicity and Efficiency: A simple robust loss ($L_q$) combined with contrastive pre-training can outperform complex multi-stage SSL pipelines like DivideMix.

Limitations and Future Work

While effective, the method shows less consistent dominance in cases of asymmetric noise (e.g., "Bird" labeled as "Airplane"), likely due to class-specific feature overlaps. Additionally, the computational cost of contrastive learning, which requires large batch sizes and long pre-training, remains a hurdle. Future work may explore more efficient ways to decouple representation learning from label noise for "plug-and-play" robustness in real-world tasks.

Summary updated

Abstract

Deep neural network-based classifiers trained with the categorical cross-entropy (CCE) loss are sensitive to label noise in the training data. One common type of method that can mitigate the impact of label noise can be viewed as supervised robust methods; one can simply replace the CCE loss with a loss that is robust to label noise, or re-weight training samples and down-weight those with higher loss values. Recently, another type of method using semi-supervised learning (SSL) has been proposed, which augments these supervised robust methods to exploit (possibly) noisy samples more effectively. Although supervised robust methods perform well across different data types, they have been shown to be inferior to the SSL methods on image classification tasks under label noise. Therefore, it remains to be seen that whether these supervised robust methods can also perform well if they can utilize the unlabeled samples more effectively. In this paper, we show that by initializing supervised robust methods using representations learned through contrastive learning leads to significantly improved performance under label noise. Surprisingly, even the simplest method (training a classifier with the CCE loss) can outperform the state-of-the-art SSL method by more than 50\% under high label noise when initialized with contrastive learning. Our implementation will be publicly available at {\url{https://github.com/arghosh/noisy_label_pretrain}}.
BibTeX
@inproceedings{ghosh2021contrastive,
  author = {Aritra Ghosh and Andrew S. Lan},
  title = {{Contrastive Learning Improves Model Robustness Under Label Noise}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=K9bL6BnhLq},
}
Figure