Abstract
Training deep neural networks on noisy datasets often leads to the memorization of incorrect labels, hindering generalization. We present ProSelfLC, an end-to-end framework for progressive self label correction. ProSelfLC adaptively adjusts training targets by incorporating the model's own predictions based on a time-dependent trust schedule and a local uncertainty measure. By employing temperature annealing to push the model toward a low-entropy state, ProSelfLC effectively corrects noisy labels during training. Our method demonstrates state-of-the-art performance across various benchmarks, including computer vision and protein sequence classification.
Introduction
The performance of deep learning models is heavily dependent on the quality of labeled data. In practice, large-scale datasets often contain "noise"—mislabeled instances that can lead models to memorize errors. Standard training using one-hot labels forces models to fit these errors. While techniques like Label Smoothing (LS) and Confidence Penalty (CP) mitigate overfitting by softening targets, they often penalize correct high-confidence predictions.
Self Label Correction (Self LC) offers a more dynamic approach by allowing the model to revise labels using its own internal knowledge. However, the primary challenge lies in determining the optimal timing and degree of trust to place in the model's predictions. ProSelfLC addresses this by introducing a progressive trust mechanism and temperature annealing to refine labels on the fly.
Methodology
ProSelfLC is an end-to-end method that modifies the target distribution $\tilde{\mathbf{q}}$ as a convex combination of the original label $\mathbf{q}$ and the model's prediction $\mathbf{p}$. The core of the approach rests on two pillars:
1. Progressive Trust ($\epsilon$)
The degree of trust $\epsilon$ is determined by both global and local factors.
- Global Trust: A sigmoid-shaped function of training time ensures the model trusts itself more as it matures.
- Local Trust: This is based on the entropy (uncertainty) of the current prediction. If the model is uncertain about a specific sample, it relies more on the original label.
2. Annealed Temperature (AT)
To prevent the model from remaining in a state of high entropy, ProSelfLC applies a low softmax temperature ($T < 1$). This "sharpening" effect pushes the model toward a low-entropy state, effectively redefining noisy targets into more confident, accurate labels.
As shown in the figure above, ProSelfLC can be implemented in both self-correction (using the target learner's own predictions) and non-self correction (using an auxiliary learner) settings, though the self-correction variant is more efficient.
Experiments and Results
The effectiveness of ProSelfLC was evaluated on several datasets, including CIFAR-100 with synthetic noise, real-world noisy datasets like Clothing1M and Food-101N, and the DeepLoc protein sequence dataset.
Performance Analysis
ProSelfLC consistently outperformed standard Cross-Entropy and other regularization baselines. For instance, on CIFAR-100 with 40% noise, ProSelfLC achieved 68.4% accuracy, significantly higher than the 47.6% achieved by standard training. On the real-world Clothing1M dataset, it reached a benchmark of 74.48% accuracy.
The training curves in the figure above illustrate that ProSelfLC maintains high accuracy on clean subsets while progressively improving performance on noisy subsets. A key observation is that the model's confidence serves as a reliable indicator of label correctness early in training, which ProSelfLC exploits to avoid memorizing noise.
Conclusion
ProSelfLC provides a robust, end-to-end solution for training deep networks on noisy data. By rewarding confidence and progressively trusting the model's internal knowledge, it transforms noisy labels into meaningful targets. Future work includes automating hyperparameter selection and extending the method to more complex structured prediction tasks.
Summary updated