Abstract
The success of deep neural networks relies heavily on large-scale, accurately labeled datasets. However, annotating such data is labor-intensive, whereas collecting web-scraped images with noisy labels is significantly more efficient. This paper addresses the challenge of training deep networks using these noisy labels. Existing methods typically attempt to correct erroneous labels or re-weight samples, but these strategies often introduce further errors. We propose a two-stage framework that instead identifies a high-confidence subset of correctly labeled images and treats the remainder as unlabeled data. By training in a semi-supervised manner, the model avoids being misled by incorrect labels while still benefiting from the full dataset's visual information. Experiments on MNIST, CIFAR-10, and Clothing1M demonstrate the effectiveness of this approach, particularly in high-noise regimes.
Introduction
Deep learning's dependence on massive datasets like ImageNet has led to the widespread use of web-scraped data. Unfortunately, web data is inherently noisy; for instance, a search for a specific category may return irrelevant images labeled with that category. Traditional noise-handling techniques, such as label correction or loss re-weighting, are prone to error propagation. If the correction mechanism is inaccurate, the model may be driven toward a poor local optimum.
As shown in Figure 1, we propose a shift from correction to identification. By isolating a "clean" seed set and ignoring the labels of ambiguous samples, we can leverage semi-supervised learning to maintain high model performance despite significant initial noise.
Methodology: The Two-Stage Framework
The proposed framework operates in two distinct stages to maximize the utility of noisy datasets while minimizing the impact of incorrect labels.
Stage 1: Mining Likely Correct Labels
The goal is to extract a subset of the training data where labels are highly likely to be correct. If a small clean validation set is available, binary classifiers are trained to distinguish between clean and noisy samples. In the absence of clean data, a baseline model is trained on the full noisy set. Samples are then selected based on the consistency between the model's predictions and the original labels, filtered by a high-confidence threshold (e.g., >90%).
Stage 2: Semi-Supervised Learning
The samples identified in Stage 1 retain their labels, while the rest are stripped of labels. The model is then trained using consistency regularization techniques, such as the $\Pi$ model or Temporal Ensembling. This process encourages the network to produce consistent outputs for the same image under different stochastic augmentations, allowing it to learn robust features from the unlabeled data guided by the high-purity labeled subset.
Experimental Results
The framework was evaluated across three benchmarks to verify its robustness:
- MNIST and CIFAR-10: In synthetic noise tests, the method significantly outperformed standard cross-entropy training. For example, at a 60% noise rate on CIFAR-10, the model maintained stability while baselines collapsed, often improving accuracy by 20-30%.
- Clothing1M: On this real-world dataset of 1 million images with approximately 39% noise, the two-stage approach achieved 79.38% accuracy. This represents a nearly 10% improvement over previous state-of-the-art methods (69.84%), demonstrating the scalability and practical utility of the framework for web-scale data.
Conclusion
This research highlights the "less is more" principle in noisy label learning. The authors found that keeping only a small fraction (e.g., ~16%) of labels can result in a near-zero error rate for the labeled set, which is far more valuable for training than a larger, corrupted set. By prioritizing the purity of the labeled set and treating ambiguous data as unlabeled, models can achieve superior robustness. Future work will focus on improving the efficiency of the label mining stage and optimizing the computational overhead associated with semi-supervised training on massive datasets.
Summary updated