Abstract
To train robust deep neural networks (DNNs), we systematically study several target modification approaches, which include output regularisation, self and non-self label correction (LC). Two key issues are discovered: (1) Self LC is the most appealing as it exploits its own knowledge and requires no extra models. However, how to automatically decide the trust degree of a learner as training goes is not well answered in the literature? (2) Some methods penalise while the others reward low-entropy predictions, prompting us to ask which one is better? To resolve the first issue, taking two well-accepted propositions—deep neural networks learn meaningful patterns before fitting noise and minimum entropy regularisation principle—we propose a novel end-to-end method named ProSelfLC, which is designed according to learning time and entropy. Specifically, given a data point, we progressively increase trust in its predicted label distribution versus its annotated one if a model has been trained for enough time and the prediction is of low entropy (high confidence). For the second issue, according to ProSelfLC, we empirically prove that it is better to redefine a meaningful low-entropy status and optimise the learner toward it. This serves as a defence of entropy minimisation. We demonstrate the effectiveness of ProSelfLC through extensive experiments in both clean and noisy settings.
Introduction
Deep Neural Networks (DNNs) are notoriously "brittle" when it comes to label noise. If a dataset contains mislabeled images (e.g., a picture of a cat labeled as a "truck"), a standard model will eventually memorize that error, hurting its ability to generalize to new data.
Existing solutions generally fall into two camps: Output Regularization (OR), like Label Smoothing, which "softens" labels to prevent overconfidence, and Label Correction (LC), which attempts to fix the labels. While "Non-self LC" uses a pre-trained teacher model, it is computationally expensive. "Self LC" uses the model's own predictions but often relies on a fixed trust score that doesn't account for the model's growth during training. ProSelfLC addresses this by asking: How can we automatically decide how much to trust a learner's own knowledge at any given point in time?
Methods: Progressive Self-Correction
ProSelfLC (Progressive Self Label Correction) modifies the target label $\tilde{q}$ by combining the original one-hot label $q$ and the model's current prediction $p$:
$$\tilde{q} = (1 - \epsilon)q + \epsilon p$$
The core innovation is the dynamic trust score $\epsilon$, calculated as the product of two components: $\epsilon = g(t) \times l(p)$.
- Global Trust Score $g(t)$: A time-dependent function that increases as training iterations $t$ grow, reflecting the observation that DNNs learn simple, meaningful patterns before memorizing noise.
- Local Trust Score $l(p)$: A data-dependent score based on entropy. If the model’s prediction for a specific image has low entropy (high confidence), $l(p)$ is high.
By multiplying these, ProSelfLC only trusts its own prediction if it has been training for a sufficient duration and is highly confident about that specific data point. This allows the model to eventually "overrule" a noisy human label if it consistently sees evidence to the contrary.
Experimental Results
The researchers evaluated ProSelfLC across several benchmarks, including CIFAR-100, ImageNet 2012, and the real-world noisy Clothing 1M dataset.
Key findings include:
- Robustness to Noise: On CIFAR-100 with 40% asymmetric noise, ProSelfLC achieved 67.9% accuracy, significantly higher than standard Cross-Entropy (59.5%).
- Defense of Entropy Minimization: While some methods penalize high confidence to prevent overfitting, ProSelfLC argues that rewarding confidence—when earned over time—is superior. As shown in the entropy plots, ProSelfLC effectively drives the model toward meaningful low-entropy states.
- Semantic Correction: The method can effectively flip the semantic class of a label on the fly, cleaning the dataset during training.
Conclusion
ProSelfLC provides a simple, end-to-end framework for training robust DNNs without the need for auxiliary teacher models. By progressively trusting its own confident predictions, the model can effectively filter out label noise and improve generalization. Future work may explore integrating this self-correction mechanism into broader knowledge distillation frameworks to further push the boundaries of model robustness.
Summary updated