A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels

Yifan Ding, Liqiang Wang, Deliang Fan, Boqing Gong

Abstract

The success of deep neural networks relies heavily on large-scale, accurately labeled datasets. However, annotating such data is labor-intensive, whereas collecting web-scraped images with noisy labels is significantly more efficient. This paper addresses the challenge of training deep networks using these noisy labels. Existing methods typically attempt to correct erroneous labels or re-weight samples, but these strategies often introduce further errors. We propose a two-stage framework that instead identifies a high-confidence subset of correctly labeled images and treats the remainder as unlabeled data. By training in a semi-supervised manner, the model avoids being misled by incorrect labels while still benefiting from the full dataset's visual information. Experiments on MNIST, CIFAR-10, and Clothing1M demonstrate the effectiveness of this approach, particularly in high-noise regimes.

Introduction

Deep learning's dependence on massive datasets like ImageNet has led to the widespread use of web-scraped data. Unfortunately, web data is inherently noisy; for instance, a search for a specific category may return irrelevant images labeled with that category. Traditional noise-handling techniques, such as label correction or loss re-weighting, are prone to error propagation. If the correction mechanism is inaccurate, the model may be driven toward a poor local optimum.

As shown in Figure 1, we propose a shift from correction to identification. By isolating a "clean" seed set and ignoring the labels of ambiguous samples, we can leverage semi-supervised learning to maintain high model performance despite significant initial noise.

Conceptual comparison between traditional noisy label handling and the proposed semi-supervised approach.

Methodology: The Two-Stage Framework

The proposed framework operates in two distinct stages to maximize the utility of noisy datasets while minimizing the impact of incorrect labels.

Stage 1: Mining Likely Correct Labels

The goal is to extract a subset of the training data where labels are highly likely to be correct. If a small clean validation set is available, binary classifiers are trained to distinguish between clean and noisy samples. In the absence of clean data, a baseline model is trained on the full noisy set. Samples are then selected based on the consistency between the model's predictions and the original labels, filtered by a high-confidence threshold (e.g., >90%).

Stage 2: Semi-Supervised Learning

The samples identified in Stage 1 retain their labels, while the rest are stripped of labels. The model is then trained using consistency regularization techniques, such as the $\Pi$ model or Temporal Ensembling. This process encourages the network to produce consistent outputs for the same image under different stochastic augmentations, allowing it to learn robust features from the unlabeled data guided by the high-purity labeled subset.

Detailed architecture of the two-stage framework, covering label mining (Stage 1) and semi-supervised training (Stage 2).

Experimental Results

The framework was evaluated across three benchmarks to verify its robustness:

  • MNIST and CIFAR-10: In synthetic noise tests, the method significantly outperformed standard cross-entropy training. For example, at a 60% noise rate on CIFAR-10, the model maintained stability while baselines collapsed, often improving accuracy by 20-30%.
  • Clothing1M: On this real-world dataset of 1 million images with approximately 39% noise, the two-stage approach achieved 79.38% accuracy. This represents a nearly 10% improvement over previous state-of-the-art methods (69.84%), demonstrating the scalability and practical utility of the framework for web-scale data.

Conclusion

This research highlights the "less is more" principle in noisy label learning. The authors found that keeping only a small fraction (e.g., ~16%) of labels can result in a near-zero error rate for the labeled set, which is far more valuable for training than a larger, corrupted set. By prioritizing the purity of the labeled set and treating ambiguous data as unlabeled, models can achieve superior robustness. Future work will focus on improving the efficiency of the label mining stage and optimizing the computational overhead associated with semi-supervised training on massive datasets.

Summary updated

Abstract

The recent success of deep neural networks is powered in part by large-scale well-labeled training data. However, it is a daunting task to laboriously annotate an ImageNet-like dateset. On the contrary, it is fairly convenient, fast, and cheap to collect training images from the Web along with their noisy labels. This signifies the need of alternative approaches to training deep neural networks using such noisy labels. Existing methods tackling this problem either try to identify and correct the wrong labels or reweigh the data terms in the loss function according to the inferred noisy rates. Both strategies inevitably incur errors for some of the data points. In this paper, we contend that it is actually better to ignore the labels of some of the data points than to keep them if the labels are incorrect, especially when the noisy rate is high. After all, the wrong labels could mislead a neural network to a bad local optimum. We suggest a two-stage framework for the learning from noisy labels. In the first stage, we identify a small portion of images from the noisy training set of which the labels are correct with a high probability. The noisy labels of the other images are ignored. In the second stage, we train a deep neural network in a semi-supervised manner. This framework effectively takes advantage of the whole training set and yet only a portion of its labels that are most likely correct. Experiments on three datasets verify the effectiveness of our approach especially when the noisy rate is high.
BibTeX
@inproceedings{ding2018semi,
  author = {Yifan Ding and Liqiang Wang and Deliang Fan and Boqing Gong},
  title = {{A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  doi = {10.1109/wacv.2018.00138},
}
Figure