Abstract
Training deep learning models on datasets with noisy labels typically leads to poor performance due to the memorization of errors. This paper demonstrates that pre-training models using self-supervised contrastive learning (SimCLR) provides a robust foundation for feature representation, enabling standard supervised methods to achieve significant performance gains. In high-noise scenarios, this approach outperforms complex state-of-the-art algorithms by substantial margins, highlighting the importance of decoupling feature learning from label handling.
Introduction
Deep neural networks are highly sensitive to label noise. When trained with standard Categorical Cross-Entropy (CCE) loss, they tend to memorize incorrect labels, which degrades generalization. Existing solutions generally fall into two categories: supervised robust methods (e.g., specialized loss functions like $L_q$) and complex semi-supervised learning (SSL) pipelines that treat noisy samples as unlabeled. While SSL methods currently lead in performance, they are often difficult to implement. This research investigates whether the failure of simpler supervised methods is due to the noise itself or a failure to learn quality visual representations when labels are unreliable.
Methodology
The authors propose a two-stage pipeline to decouple representation learning from label noise:
- Self-Supervised Pre-training: Using the SimCLR framework, the model learns to recognize augmented versions of the same image without using any labels. This ensures that the initial feature learning is not corrupted by label noise.
- Fine-tuning with Robust Methods: The pre-trained model is fine-tuned using supervised methods, including standard CCE, $L_q$ loss, and Meta-Weight Net (MWNet).
Experimental Results
The approach was evaluated on CIFAR-10, CIFAR-100, and the real-world Clothing1M dataset.
Performance in Extreme Noise
The most significant gains were observed in high-noise environments. On CIFAR-100 with 90% symmetric noise, standard CCE with random initialization achieves only 10.1% accuracy. However, initializing with SimCLR pre-training boosts this to 52.11%, a significant relative gain over the previous state-of-the-art, DivideMix (31.5%). Similarly, on CIFAR-10 with 90% noise, SimCLR pre-training improves CCE performance to 82.9%, surpassing DivideMix's 76%.
Real-World Application
On the Clothing1M dataset, which contains naturally occurring noise from web-scraped data, the SimCLR-initialized models showed strong performance, demonstrating the practical utility of this strategy beyond synthetic benchmarks.
Key Insights
- The Representation Bottleneck: Supervised robust methods struggle primarily because they fail to build a solid feature foundation on noisy data. Contrastive pre-training provides this foundation, making even "fragile" losses like CCE resilient.
- Decoupling is Key: Separating the learning of visual features from the learning of class labels prevents the model from memorizing noise during critical early training stages.
- Simplicity and Efficiency: A simple robust loss ($L_q$) combined with contrastive pre-training can outperform complex multi-stage SSL pipelines like DivideMix.
Limitations and Future Work
While effective, the method shows less consistent dominance in cases of asymmetric noise (e.g., "Bird" labeled as "Airplane"), likely due to class-specific feature overlaps. Additionally, the computational cost of contrastive learning, which requires large batch sizes and long pre-training, remains a hurdle. Future work may explore more efficient ways to decouple representation learning from label noise for "plug-and-play" robustness in real-world tasks.
Summary updated