Contrastive Learning Improves Model Robustness Under Label Noise

Aritra Ghosh, Andrew S. Lan

Abstract

Deep neural network-based classifiers trained with the categorical cross-entropy (CCE) loss are sensitive to label noise in the training data. One common type of method that can mitigate the impact of label noise can be viewed as supervised robust methods; one can simply replace the CCE loss with a loss that is robust to label noise, or re-weight training samples and down-weight those with higher loss values. Recently, another type of method using semi-supervised learning (SSL) has been proposed, which augments these supervised robust methods to exploit (possibly) noisy samples more effectively. Although supervised robust methods perform well across different data types, they have been shown to be inferior to the SSL methods on image classification tasks under label noise. Therefore, it remains to be seen that whether these supervised robust methods can also perform well if they can utilize the unlabeled samples more effectively. In this paper, we show that by initializing supervised robust methods using representations learned through contrastive learning leads to significantly improved performance under label noise. Surprisingly, even the simplest method (training a classifier with the CCE loss) can outperform the state-of-the-art SSL method by more than 50% under high label noise when initialized with contrastive learning.

Introduction

In real-world datasets, labels are rarely perfect. When deep neural networks (DNNs) are trained using the standard Categorical Cross-Entropy (CCE) loss, they tend to memorize these noisy labels, destroying their ability to generalize to clean data. Historically, researchers tackled this with supervised robust methods, such as designing noise-tolerant loss functions or re-weighting training samples to ignore likely errors. Recently, however, Semi-Supervised Learning (SSL) methods have dominated the field by treating noisy samples as unlabeled data to learn better visual features.

The performance gap between these two paradigms on image datasets raises a fundamental question: Do supervised robust methods fail because they are inherently bad at handling noise, or do they fail because they cannot learn high-quality visual representations when clean data is scarce? This research demonstrates that representation learning is the primary bottleneck.

Methodology

To test the hypothesis that representation is the key, the authors decouple representation learning from the noisy classification task through a two-step process:

  1. Self-Supervised Pre-training: Instead of starting with random weights or ImageNet pre-training, the base encoder (ResNet-50) is initialized using SimCLR. This contrastive learning framework learns visual representations by comparing augmented views of the same image, completely ignoring the noisy labels during this phase.
  2. Robust Fine-Tuning: Once the network understands the visual structure of the data, the classification head is fine-tuned using standard supervised robust methods, including standard CCE loss, Generalized Cross-Entropy ($L_q$) loss, and Meta-Weight Net (MWNet).

Experimental Results

Synthetic Noise Benchmarks

The authors evaluated their approach on CIFAR-10 and CIFAR-100 with varying levels of symmetric and asymmetric noise. The results under extreme symmetric noise are particularly striking.

Performance comparison on CIFAR-10 and CIFAR-100 under varying levels of symmetric noise.

As shown in the table above, with standard random initialization, the CCE loss drops to a dismal 10.1% accuracy on CIFAR-100 under 90% noise. However, with SimCLR initialization, CCE jumps to 52.11%, significantly outperforming the previous state-of-the-art SSL method, DivideMix (31.5%). Furthermore, methods like MWNet, which fail to learn with random initialization at 90% noise, achieve 58.21% accuracy when paired with SimCLR pre-training.

Real-World Noise: Clothing1M

To validate the approach beyond synthetic benchmarks, the authors tested it on Clothing1M, which contains 1 million images with ~38% natural label noise.

Accuracy on the Clothing1M dataset with natural label noise comparing different initializers.

The results indicate that self-supervision on the target dataset can be more effective than transferring weights from external clean datasets like ImageNet. SimCLR initialization improved standard CCE performance by approximately 4-6% compared to using a standard ImageNet pre-trained model, proving the technique's efficacy on real-world noisy data.

Discussion and Conclusion

The core insight of this work is that the massive performance gap between complex SSL methods and simpler robust methods is largely due to representation learning. Once provided with a strong visual foundation via contrastive learning, even "sensitive" losses like CCE perform remarkably well under noise. This suggests that researchers should explicitly decouple representation learning from label noise management. By tackling these tasks independently, simple robust methods can achieve state-of-the-art results without the complexity of multi-stage SSL pipelines.

While SimCLR initialization dominates under symmetric noise, supervised robust methods still slightly lag behind state-of-the-art SSL methods when dealing with asymmetric (class-confusing) noise. Future work may focus on designing complementary approaches that leverage these independent strengths to further improve robustness in complex, real-world scenarios.

Summary updated

Abstract

Deep neural network-based classifiers trained with the categorical cross-entropy (CCE) loss are sensitive to label noise in the training data. One common type of method that can mitigate the impact of label noise can be viewed as supervised robust methods; one can simply replace the CCE loss with a loss that is robust to label noise, or re-weight training samples and down-weight those with higher loss values. Recently, another type of method using semi-supervised learning (SSL) has been proposed, which augments these supervised robust methods to exploit (possibly) noisy samples more effectively. Although supervised robust methods perform well across different data types, they have been shown to be inferior to the SSL methods on image classification tasks under label noise. Therefore, it remains to be seen that whether these supervised robust methods can also perform well if they can utilize the unlabeled samples more effectively. In this paper, we show that by initializing supervised robust methods using representations learned through contrastive learning leads to significantly improved performance under label noise. Surprisingly, even the simplest method (training a classifier with the CCE loss) can outperform the state-of-the-art SSL method by more than 50\% under high label noise when initialized with contrastive learning. Our implementation will be publicly available at {\url{https://github.com/arghosh/noisy_label_pretrain}}.
BibTeX
@inproceedings{ghosh2021contrastive,
  author = {Aritra Ghosh and Andrew S. Lan},
  title = {{Contrastive Learning Improves Model Robustness Under Label Noise}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=K9bL6BnhLq},
}
Figure