TL;DR
This paper introduces ProSelfLC, a novel method for training robust deep neural networks in the presence of noisy labels. ProSelfLC progressively and adaptively trusts a model's own predictions as it learns, striking a balance between human annotations and the model's evolving knowledge, and demonstrating the benefit of rewarding low-entropy predictions.
What Problem Does This Paper Address?
Training deep neural networks (DNNs) often suffers from the presence of noisy labels in the training data. These incorrect labels can mislead the model, causing it to learn spurious correlations and ultimately degrade its performance and generalization capabilities. While various techniques exist to mitigate this, such as output regularization (e.g., label smoothing) and label correction (LC), they often have limitations. Specifically, self-supervised LC methods, which leverage the model's own predictions without external help, lack a clear strategy for determining how much to trust these predictions as training progresses. Furthermore, there's a debate on whether to penalize or reward low-entropy (high-confidence) predictions, with some methods penalizing them to avoid overfitting to noise, while others implicitly reward them.
Methods
ProSelfLC tackles these issues by introducing a novel end-to-end trainable method for progressive and adaptive self-label correction. The core idea is to dynamically adjust the trust placed on the model's own predictions based on two factors:
- Learning Time (Global Trust, $g(t)$): As training progresses (denoted by iteration $t$), the model's knowledge grows. ProSelfLC gradually increases its trust in the model's predictions over time. This is modeled using a sigmoid-like function that ensures trust is low in the early stages and high in the later stages of training.
- Prediction Confidence (Local Trust, $l(p)$): For each data point, ProSelfLC assesses the confidence of the model's prediction ($p$). High-confidence predictions (low entropy) are rewarded with higher trust. This is quantified using the normalized entropy of the prediction.
The final trust score, $\epsilon_{\text{ProSelfLC}}$, is a product of these two factors: $\epsilon_{\text{ProSelfLC}} = g(t) \times l(p)$. This $\epsilon$ value then determines how the target label distribution is constructed, blending the original (potentially noisy) human annotation ($q$) with the model's prediction ($p$) as: $\tilde{q}_{\text{ProSelfLC}} = (1 - \epsilon_{\text{ProSelfLC}})q + \epsilon_{\text{ProSelfLC}}p$. This approach allows the model to progressively rely more on its own learned patterns, especially when those patterns are confident and consistent with the evolving understanding of the data.
Datasets
The paper evaluates ProSelfLC on standard image classification benchmarks:
- CIFAR-100: A dataset with 100 classes, used with varying levels of synthetic label noise (asymmetric and symmetric).
- ImageNet 2012: A large-scale dataset used in a standard setting (without explicit noise injection, but likely containing inherent noise).
- Clothing 1M: A real-world dataset with a significant percentage of noisy labels, used to evaluate performance on naturally occurring label noise.
Key Insights and Ideas
- Progressive Trust is Key: The most significant insight is that a model's trustworthiness evolves during training. Early in training, human labels are more reliable; later, the model's own confident predictions become valuable. ProSelfLC formalizes this dynamic.
- Rewarding Confidence, Not Penalizing: Contrary to some methods that penalize high-confidence predictions to combat noise, ProSelfLC demonstrates that rewarding meaningful low-entropy predictions is more effective. This redefines the role of entropy minimization.
- Self-Knowledge is Powerful: By leveraging the model's own learned representations and predictions, ProSelfLC achieves state-of-the-art results without requiring external "teacher" models or complex noise transition matrix estimation.
- Unified Framework: ProSelfLC provides a unified perspective on target modification, showing how existing methods like Label Smoothing (LS) and Confidence Penalty (CP) can be seen as special cases or related to label correction.
Key Results
- State-of-the-Art Performance: ProSelfLC consistently outperforms existing methods, including CCE, LS, CP, and Boot-soft, on CIFAR-100 and ImageNet 2012 in both clean and noisy settings. For instance, on CIFAR-100 with 40% asymmetric noise, ProSelfLC achieves an accuracy of 67.9%, significantly higher than other methods.
- Robustness to Noise: The method shows remarkable robustness to varying levels of label noise, maintaining high accuracy even with substantial corruption.
- Improved Generalization: ProSelfLC leads to better generalization, as evidenced by its performance on test sets and its ability to avoid fitting noisy labels.
- Effective Semantic Class Correction: ProSelfLC effectively corrects semantic class errors in noisy labels, as visualized in learning dynamics plots.
Limitations and Future Work
- Hyperparameter Tuning: While ProSelfLC requires tuning hyperparameters like $\Gamma$ and $B$ (related to the global trust function), the paper suggests these are task-dependent and can be searched on a validation set.
- Potential for Memorization: The authors acknowledge that in rare cases, a model might still memorize a few noisy labels with high confidence, although ProSelfLC significantly mitigates this.
- Future Work: The paper suggests that ProSelfLC can be used to train better teacher models for knowledge distillation and can be integrated into stage-wise approaches for further improvements. Exploring its application in other domains beyond image classification is also a natural extension.
Summary updated