TL;DR
This paper introduces CoNL, a novel method that leverages self-supervised learning (SSL) to improve the estimation of instance-dependent noise transition matrices. By using SSL to infer cleaner labels, CoNL significantly boosts the accuracy of classifiers trained on noisy datasets, achieving state-of-the-art results.
What Problem Does This Paper Address?
Training deep learning models on large datasets is often hampered by the presence of noisy labels, which can arise from automated data collection or human annotation errors. These noisy labels can degrade model performance. A key component in addressing this is understanding the "transition matrix" – how often a true label is flipped to a specific incorrect label. However, in real-world scenarios, this matrix is not only unknown but can also vary from one instance to another (instance-dependent). Estimating this instance-dependent transition matrix is challenging because it requires knowledge of the true labels, which are precisely what's missing. Existing methods struggle with this inherent difficulty, especially when dealing with complex noise patterns.
Methods
The proposed method, CoNL (Contrastive Label-Noise Learning), tackles this problem in two main stages:
- Contrastive Co-Selecting: This stage leverages the power of self-supervised learning, specifically using a Momentum Contrast (MoCo) framework. The idea is that SSL can learn robust visual representations that are less susceptible to label noise. By training a backbone network with MoCo, the authors obtain these representations. Then, two distinct classifiers and transition matrices are trained simultaneously using these representations and the noisy labels. Crucially, these classifiers are trained with different data augmentation strategies (strong and weak) to encourage diversity in their learning abilities. This allows them to identify and select "confident" examples that are likely to have correct or near-correct labels, effectively filtering out some of the noise.
- Constraint T(x) Revision: In this stage, the selected confident examples are used to refine the estimated instance-dependent transition matrix. The core principle is that a good transition matrix should minimize classification losses on both the noisy labels and the inferred clean labels of these confident examples. By minimizing these losses, the transition matrix is iteratively improved. This process also allows for fine-tuning of the classifier head and the backbone model, further enhancing overall performance.
Datasets
The authors evaluated their method on several datasets:
-
Synthetic Datasets:
- Fashion-MNIST
- SVHN
- CIFAR-10 These datasets were corrupted with instance-dependent label noise at various rates (10% to 50%).
-
Real-world Dataset:
- CIFAR-10N This dataset contains human-annotated noisy labels with five different noise types.
-
Additional Datasets for Robustness:
- CIFAR-100
- WebVision These were used to test the method's performance with a larger number of classes and in a more challenging semi-supervised setting.
Key Insights and Ideas
- SSL as a Noise Filter: The paper demonstrates that self-supervised learning, particularly contrastive learning, is not just for representation learning but can be a powerful tool for inferring cleaner labels in the presence of noise. The learned representations are inherently more robust to label errors.
- Diversity in Classifiers for Better Selection: Training two classifiers with different augmentation strategies (strong vs. weak) is key. This diversity allows them to identify different types of confident examples, leading to a more comprehensive and reliable set of clean-label candidates.
- Iterative Refinement of Transition Matrix: The "Constraint T(x) Revision" stage highlights the benefit of using the inferred clean labels to iteratively improve the transition matrix estimation. This feedback loop is crucial for achieving high accuracy, especially under high noise levels.
- Instance-Dependent Noise is a Major Hurdle: The paper strongly emphasizes that accurately estimating instance-dependent transition matrices is significantly harder than estimating a global one, and CoNL's success in this area is a key contribution.
Key Results
CoNL achieved state-of-the-art performance across various datasets and noise levels. For instance:
- On Fashion-MNIST with 50% instance-dependent noise, CoNL reached an accuracy of 86.01%, significantly outperforming other methods.
- On SVHN with 50% noise, CoNL achieved 86.51% accuracy.
- On CIFAR-10 with 50% noise, CoNL reached 84.79% accuracy.
- The paper also shows that the "Constraint T(x) Revision" stage effectively reduces the estimation error of the transition matrix.
Limitations and Future Work
- Hyperparameter Sensitivity: While not explicitly detailed as a limitation, the method relies on hyperparameters like λ and τ for confident example selection, which might require careful tuning.
- Computational Cost: The use of MoCo for representation learning and the iterative refinement process likely increase the computational cost compared to simpler methods.
- Future Work: The paper suggests that the proposed method can be extended to other semi-supervised learning tasks where label noise is present. Further exploration into more advanced SSL techniques and more sophisticated ways to leverage the learned representations for transition matrix estimation could be beneficial.
Summary updated