Making Deep Neural Networks Robust to Label Noise: Cross-Training With a Novel Loss Function

Zhen Qin, Zhengwen Zhang, Yan Li, Jun GuoIEEE Access 2019 · OpenReview

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success in supervised learning, but their performance is heavily dependent on the quality of labeled training data. In real-world applications, datasets often contain "label noise"—incorrect labels resulting from crowdsourcing errors or automated data collection. Standard DNNs are prone to memorizing these noisy labels, which leads to overfitting and poor generalization. This research presents a robust training framework designed to mitigate these effects through a novel loss function, a cross-training strategy, and online data filtration.

The Three-Pronged Defense Against Noise

The proposed approach addresses the challenge of messy labels through three integrated components:

1. Rectified Cross-Entropy (RCE) Loss

Standard cross-entropy loss can be counterproductive when labels are incorrect, as it forces the model to minimize error even for garbage data. The authors introduce Rectified Cross-Entropy (RCE), which adaptively re-weights the loss. If the model's prediction for a label is very low—suggesting the label is likely wrong—the RCE loss assigns it a lower weight. This prevents the network from trying too hard to "learn" mislabeled samples.

2. Cross-Training Strategy

To prevent a single model from falling into a self-reinforcing loop of its own biases, the framework trains two identical models (Model A and Model B) simultaneously. These models engage in a "peer-review" process where Model A’s predictions are used to weight the loss for Model B, and vice versa. This "multi-view" approach ensures that the training curriculum for each model is guided by the independent perspective of its partner.

3. Online Data Filtration

In addition to re-weighting, the system actively cleans the training set. During training, the framework calculates the average prediction probability from both models. If this average falls below a specific threshold (typically $1/\text{number of classes}$), the sample is flagged as noisy and removed from the training set entirely, ensuring the models focus on high-quality data.

Experimental Results

The researchers evaluated the framework on several major benchmarks, including MNIST, CIFAR-10, CIFAR-100, and the real-world Clothing1M dataset.

Test accuracy over training epochs for varying noise levels (η). The framework maintains high performance and stability even as the ratio of mislabeled data increases.

As illustrated in the figure above, the framework demonstrates significant robustness across different noise scales. Even when the noise ratio ($\eta$) is as high as 0.8 (meaning 80% of labels are wrong), the model maintains a stable and useful accuracy level, avoiding the catastrophic performance drops seen in standard training methods.

Key Performance Highlights:

  • CIFAR-100 (80% Noise): The proposed method achieved 37.8% accuracy, compared to just 10.1% for standard cross-entropy training.
  • Clothing1M (Real-World Noise): The framework reached a Top-1 accuracy of 74.6%, outperforming the standard baseline of 68.9%. This confirms that the method is effective against complex, natural noise found in web-crawled data.

Conclusion

By combining adaptive loss weighting with a collaborative dual-model architecture, this research provides a powerful tool for training deep learning models in the "wild." While the dual-model approach increases computational overhead, the gains in robustness make it a compelling choice for large-scale applications where perfect labeling is impossible. Future work will focus on dynamic thresholding for data filtration to further refine the learning process.

Summary updated

Abstract

Deep neural networks (DNNs) have achieved astonishing results on a variety of supervised learning tasks owing to a large scale of well-labeled training data. However, as recent researches have pointed out, the generalization performance of DNNs is likely to sharply deteriorate when training data contains label noise. In order to address this problem, a novel loss function is proposed to guide DNNs to pay more attention to clean samples via adaptively weighing the traditional cross-entropy loss. Under the guidance of this loss function, a cross-training strategy is designed by leveraging two synergic DNN models, each of which plays the roles of both updating its own parameters and generating curriculums for the other one. In addition, this paper further proposes an online data filtration mechanism and integrates it into the final cross-training framework, which simultaneously optimizes DNN models and filters out noisy samples. The proposed approach is evaluated through a great deal of experiments on several benchmark datasets with man-made or real-world label noise, and the results have demonstrated its robustness to different noise types and noise scales.
BibTeX
@inproceedings{qin2019making,
  author = {Zhen Qin and Zhengwen Zhang and Yan Li and Jun Guo},
  title = {{Making Deep Neural Networks Robust to Label Noise: Cross-Training With a Novel Loss Function}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=0ThgRwWlET},
}
Figure