TL;DR
This paper revisits label correction methods for deep learning with noisy labels, proposing a new method called DULC. DULC uses a novel interpolation weight based on normalized Jensen-Shannon Divergence to ensure sample-wise dynamism and class-wise uniformity, leading to improved performance and simpler architecture compared to state-of-the-art methods.
What Problem Does This Paper Address?
Deep neural networks (DNNs) are highly effective but rely on large, high-quality datasets. Automatically collected datasets often contain noisy labels, which can severely degrade DNN performance. While label correction methods aim to fix these noisy labels by interpolating between the noisy label and the model's prediction, they often suffer from false corrections and lag behind more complex semi-supervised methods. The core challenge is designing an effective interpolation strategy that can accurately identify and correct mislabeled samples without introducing new errors, especially in scenarios with high noise levels and imbalanced class distributions.
Methods
The paper proposes DULC (Dynamic and Uniform Label Correction), a method that addresses the limitations of existing label correction techniques. DULC is built upon a theoretical analysis of gradient scaling, which reveals two crucial properties for an ideal interpolation weight:
- Sample-wise dynamic: The weight should adapt to each sample's "cleanliness," meaning it should be higher for potentially mislabeled samples and lower for likely clean ones.
- Class-wise uniformity: The weights should be aligned across different classes to prevent class difficulty from introducing bias into the correction process.
To achieve these properties, DULC employs:
- Jensen-Shannon Divergence (JSD) as an interpolation weight: JSD is used to quantify the difference between the model's prediction and the noisy label. Empirically, higher JSD values correlate with mislabeled samples, thus providing a dynamic weight that reflects label cleanliness.
- JSD Normalization: To ensure class-wise uniformity, the JSD values within each class are normalized using a min-max approach. This aligns the interpolation weights across different classes, mitigating biases caused by varying class difficulties.
- Augmentation and Sharpening: The paper theoretically demonstrates that sharpening predictions (making them more confident) helps in memorizing true classes. DULC achieves this by using a "weak and strong augmentation" strategy, where the model is trained on strong augmentations while label correction is performed using predictions from weak augmentations. A sharpening function with a temperature coefficient is also applied.
- Linear Decay Strategy: A linear decay strategy is used to constrain the interpolation weight in the early stages of training, allowing for a smoother transition and retaining more information about assigned labels.
Datasets
The paper evaluates DULC on a variety of datasets, including:
- Simulated noisy datasets:
- CIFAR-10
- CIFAR-100
- TinyImageNet
- Real-world noisy datasets:
- WebVision
- Clothing1M
These datasets cover different noise types (symmetric and asymmetric) and noise ratios, providing a comprehensive evaluation of DULC's robustness.
Key Insights and Ideas
- Theoretical grounding for interpolation weights: The paper provides a theoretical analysis of gradient scaling, offering a principled justification for the need for sample-wise dynamic and class-wise uniform interpolation weights in label correction.
- JSD as a proxy for label cleanliness: The Jensen-Shannon Divergence is shown to be an effective metric for quantifying the difference between predictions and noisy labels, naturally providing a dynamic weight that aligns with label cleanliness.
- Class-wise uniformity is crucial for hard classes: The paper highlights that without class-wise uniformity, methods can struggle with difficult classes, leading to cascading errors and performance degradation, especially in high noise scenarios.
- Sharpening predictions aids true class memorization: Theoretical evidence suggests that sharpening predictions, rather than just correcting labels, can further improve the memorization of correct labels.
Key Results
DULC consistently outperforms state-of-the-art (SOTA) label correction and sample selection methods across various datasets and noise levels.
- CIFAR-10/100: DULC achieves significant improvements in test accuracy, often outperforming baselines by a considerable margin, especially under high noise ratios. For example, on CIFAR-10 with 90% symmetric noise, DULC achieves 93.5% accuracy, while other methods struggle to reach much lower scores.
- TinyImageNet: DULC demonstrates competitive performance, achieving the best or suboptimal results on all noise ratios, even with a simpler architecture.
- WebVision and Clothing1M: DULC achieves superior performance on these real-world noisy datasets, outperforming baselines and demonstrating its effectiveness in complex, unbalanced scenarios.
- Ablation studies: The studies confirm the importance of each component of DULC, showing significant performance drops when augmentation, JSD metric, JSD normalization, or linear decay are removed.
Limitations and Future Work
- TinyImageNet performance: While DULC performs well on TinyImageNet, the paper notes that selection-based methods might be slightly more robust in this specific scenario due to the dataset's characteristics (more classes, fewer samples per class), which can lead to increased prediction confusion in early training stages.
- Future work: The paper suggests that DULC can be combined with other techniques like MixUp augmentation and contrastive learning loss to further mitigate noisy label memorization, indicating potential for future research in hybrid approaches. The theoretical analysis also opens avenues for exploring other metrics or normalization strategies for interpolation weights.
Summary updated