ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks

Xinshao Wang, Hua Yang, Elyor Kodirov, David A. Clifton, Neil M. Robertson

Abstract

To train robust deep neural networks (DNNs), we systematically study several target modification approaches, which include output regularisation, self and non-self label correction (LC). Two key issues are discovered: (1) Self LC is the most appealing as it exploits its own knowledge and requires no extra models. However, how to automatically decide the trust degree of a learner as training goes is not well answered in the literature? (2) Some methods penalise while the others reward low-entropy predictions, prompting us to ask which one is better? To resolve the first issue, taking two well-accepted propositions—deep neural networks learn meaningful patterns before fitting noise and minimum entropy regularisation principle—we propose a novel end-to-end method named ProSelfLC, which is designed according to learning time and entropy. Specifically, given a data point, we progressively increase trust in its predicted label distribution versus its annotated one if a model has been trained for enough time and the prediction is of low entropy (high confidence). For the second issue, according to ProSelfLC, we empirically prove that it is better to redefine a meaningful low-entropy status and optimise the learner toward it. This serves as a defence of entropy minimisation. We demonstrate the effectiveness of ProSelfLC through extensive experiments in both clean and noisy settings.

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success across various domains, yet they remain notoriously sensitive to "noisy" labels—training data where annotations are incorrect. Standard training objectives, such as Cross-Entropy, often lead models to eventually "memorize" these errors, which significantly degrades their ability to generalize to unseen data.

Existing solutions typically involve output regularization (like Label Smoothing) or Label Correction (LC). While Label Smoothing prevents over-confidence, it ignores the model's growing internal knowledge. Conversely, traditional Label Correction often requires expensive auxiliary "teacher" models (Non-self LC). This research introduces ProSelfLC (Progressive Self Label Correction), an end-to-end framework that allows a model to automatically decide how much to trust its own predictions over potentially noisy training labels as learning progresses.

Methodology: Progressive Self Label Correction

ProSelfLC modifies the target label ($\tilde{q}$) by blending the original annotated label ($q$) with the model’s current prediction ($p$):

$$\tilde{q} = (1 - \epsilon)q + \epsilon p$$

The Label Correction (LC) framework: Self LC vs. Non-self LC.

The core innovation lies in the adaptive calculation of the trust score $\epsilon$. Unlike previous methods using fixed values, ProSelfLC calculates $\epsilon$ using two dynamic components:

  1. Global Trust Score $g(t)$: A time-dependent function that increases as training progresses, leveraging the observation that DNNs learn simple, meaningful patterns early before fitting noise.
  2. Local Trust Score $l(p)$: A data-dependent score based on entropy. If the model is highly confident (low entropy) about a specific data point, $l(p)$ is high.

By defining $\epsilon = g(t) \times l(p)$, the model only corrects a label if it has been trained for a sufficient duration and is confident about that specific example. This progressive nature prevents early "confirmation bias" where a model might reinforce its own initial mistakes.

Experimental Results

The researchers evaluated ProSelfLC on several benchmarks, including CIFAR-100 (with synthetic noise), ImageNet 2012, and the real-world noisy dataset Clothing 1M (~38.46% natural noise).

Performance comparison and training dynamics of ProSelfLC against baselines.

Key findings from the experiments include:

  • Superior Robustness: On CIFAR-100 with 40% asymmetric noise, ProSelfLC achieved 67.9% accuracy, significantly outperforming standard Cross-Entropy (59.5%) and existing self-correction methods like Boot-soft (61.0%).
  • Real-World Efficacy: On the Clothing 1M dataset, ProSelfLC reached 73.4% accuracy, surpassing specialized noise-handling baselines.
  • No Clean Data Penalty: Unlike some robust methods that sacrifice performance on clean data, ProSelfLC slightly improved ImageNet performance (76.0% vs 75.5% for standard training), acting as an effective regularizer even for correct labels.

Discussion and Conclusion

ProSelfLC demonstrates that a model's internal knowledge is a valuable, "free" resource for self-improvement. By redefining the target toward a meaningful low-entropy status, the paper provides a strong defense for the entropy minimization principle in deep learning. While the method requires some hyperparameter tuning for growth speed ($B$), it offers a robust, end-to-end solution for training DNNs in the presence of label noise without the overhead of auxiliary models. Future work may explore combining ProSelfLC with non-self knowledge distillation to further push the boundaries of robust representation learning.

Summary updated

Abstract

To train robust deep neural networks (DNNs), we systematically study several target modification approaches, which include output regularisation, self and non-self label correction (LC). Two key issues are discovered: (1) Self LC is the most appealing as it exploits its own knowledge and requires no extra models. However, how to automatically decide the trust degree of a learner as training goes is not well answered in the literature? (2) Some methods penalise while the others reward low-entropy predictions, prompting us to ask which one is better? To resolve the first issue, taking two well-accepted propositions--deep neural networks learn meaningful patterns before fitting noise [3] and minimum entropy regularisation principle [10]--we propose a novel end-to-end method named ProSelfLC, which is designed according to learning time and entropy. Specifically, given a data point, we progressively increase trust in its predicted label distribution versus its annotated one if a model has been trained for enough time and the prediction is of low entropy (high confidence). For the second issue, according to ProSelfLC, we empirically prove that it is better to redefine a meaningful low-entropy status and optimise the learner toward it. This serves as a defence of entropy minimisation. We demonstrate the effectiveness of ProSelfLC through extensive experiments in both clean and noisy settings. The source code is available at https://github.com/XinshaoAmosWang/ProSelfLC-CVPR2021. Keywords: entropy minimisation, maximum entropy, confidence penalty, self knowledge distillation, label correction, label noise, semi-supervised learning, output regularisation
BibTeX
@inproceedings{wang2021proselflc,
  author = {Xinshao Wang and Hua Yang and Elyor Kodirov and David A. Clifton and Neil M. Robertson},
  title = {{ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=IWpPX_SylI},
}
Figure