ProSelfLC: Progressive Self Label Correction Towards A Low-Temperature Entropy State

Xinshao Wang, Yang Hua, Elyor Kodirov, Sankha Subhra Mukherjee, David A. Clifton, Neil M. RobertsonarXiv 2022 · DOI · arXiv

Abstract

Training deep neural networks on noisy datasets often leads to the memorization of incorrect labels, hindering generalization. We present ProSelfLC, an end-to-end framework for progressive self label correction. ProSelfLC adaptively adjusts training targets by incorporating the model's own predictions based on a time-dependent trust schedule and a local uncertainty measure. By employing temperature annealing to push the model toward a low-entropy state, ProSelfLC effectively corrects noisy labels during training. Our method demonstrates state-of-the-art performance across various benchmarks, including computer vision and protein sequence classification.

Introduction

The performance of deep learning models is heavily dependent on the quality of labeled data. In practice, large-scale datasets often contain "noise"—mislabeled instances that can lead models to memorize errors. Standard training using one-hot labels forces models to fit these errors. While techniques like Label Smoothing (LS) and Confidence Penalty (CP) mitigate overfitting by softening targets, they often penalize correct high-confidence predictions.

Self Label Correction (Self LC) offers a more dynamic approach by allowing the model to revise labels using its own internal knowledge. However, the primary challenge lies in determining the optimal timing and degree of trust to place in the model's predictions. ProSelfLC addresses this by introducing a progressive trust mechanism and temperature annealing to refine labels on the fly.

Methodology

ProSelfLC is an end-to-end method that modifies the target distribution $\tilde{\mathbf{q}}$ as a convex combination of the original label $\mathbf{q}$ and the model's prediction $\mathbf{p}$. The core of the approach rests on two pillars:

1. Progressive Trust ($\epsilon$)

The degree of trust $\epsilon$ is determined by both global and local factors.

  • Global Trust: A sigmoid-shaped function of training time ensures the model trusts itself more as it matures.
  • Local Trust: This is based on the entropy (uncertainty) of the current prediction. If the model is uncertain about a specific sample, it relies more on the original label.

2. Annealed Temperature (AT)

To prevent the model from remaining in a state of high entropy, ProSelfLC applies a low softmax temperature ($T < 1$). This "sharpening" effect pushes the model toward a low-entropy state, effectively redefining noisy targets into more confident, accurate labels.

Comparison of Self and Non-self Label Correction mechanisms.

As shown in the figure above, ProSelfLC can be implemented in both self-correction (using the target learner's own predictions) and non-self correction (using an auxiliary learner) settings, though the self-correction variant is more efficient.

Experiments and Results

The effectiveness of ProSelfLC was evaluated on several datasets, including CIFAR-100 with synthetic noise, real-world noisy datasets like Clothing1M and Food-101N, and the DeepLoc protein sequence dataset.

Performance Analysis

ProSelfLC consistently outperformed standard Cross-Entropy and other regularization baselines. For instance, on CIFAR-100 with 40% noise, ProSelfLC achieved 68.4% accuracy, significantly higher than the 47.6% achieved by standard training. On the real-world Clothing1M dataset, it reached a benchmark of 74.48% accuracy.

Training dynamics showing accuracy and confidence across clean and noisy subsets at different noise ratios.

The training curves in the figure above illustrate that ProSelfLC maintains high accuracy on clean subsets while progressively improving performance on noisy subsets. A key observation is that the model's confidence serves as a reliable indicator of label correctness early in training, which ProSelfLC exploits to avoid memorizing noise.

Conclusion

ProSelfLC provides a robust, end-to-end solution for training deep networks on noisy data. By rewarding confidence and progressively trusting the model's internal knowledge, it transforms noisy labels into meaningful targets. Future work includes automating hyperparameter selection and extending the method to more complex structured prediction tasks.

Summary updated

Abstract

There is a family of label modification approaches including self and non-self label correction (LC), and output regularisation. They are widely used for training robust deep neural networks (DNNs), but have not been mathematically and thoroughly analysed together. We study them and discover three key issues: (1) We are more interested in adopting Self LC as it leverages its own knowledge and requires no auxiliary models. However, it is unclear how to adaptively trust a learner as the training proceeds. (2) Some methods penalise while the others reward low-entropy (i.e., high-confidence) predictions, prompting us to ask which one is better. (3) Using the standard training setting, a learned model becomes less confident when severe noise exists. Self LC using high-entropy knowledge would generate high-entropy targets. To resolve the issue (1), inspired by a well-accepted finding, i.e., deep neural networks learn meaningful patterns before fitting noise, we propose a novel end-to-end method named ProSelfLC, which is designed according to the learning time and prediction entropy. Concretely, for any data point, we progressively and adaptively trust its predicted probability distribution versus its annotated one if a network has been trained for a relatively long time and the prediction is of low entropy. For the issue (2), the effectiveness of ProSelfLC defends entropy minimisation. By ProSelfLC, we empirically prove that it is more effective to redefine a semantic low-entropy state and optimise the learner toward it. To address the issue (3), we decrease the entropy of self knowledge using a low temperature before exploiting it to correct labels, so that the revised labels redefine low-entropy target probability distributions. We demonstrate the effectiveness of ProSelfLC through extensive experiments in both clean and noisy settings, and on both image and protein datasets.
BibTeX
@misc{wang2022proselflc,
  author = {Xinshao Wang and Yang Hua and Elyor Kodirov and Sankha Subhra Mukherjee and David A. Clifton and Neil M. Robertson},
  title = {{ProSelfLC: Progressive Self Label Correction Towards A Low-Temperature Entropy State}},
  year = {2022},
  eprint = {2207.00118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.00118},
}
Figure