Improving the Estimation of Instance-dependent Transition Matrix by using Self-supervised Learning

Yexiong Lin, Yu Yao, Zhaoqing Wang, Xu Shen, Jun Yu, Bo Han, Tongliang LiuICLR 2023 · OpenReview

TL;DR

This paper introduces CoNL, a novel method that leverages self-supervised learning (SSL) to improve the estimation of instance-dependent noise transition matrices. By using SSL to infer cleaner labels, CoNL significantly boosts the accuracy of classifiers trained on noisy datasets, achieving state-of-the-art results.

What Problem Does This Paper Address?

Training deep learning models on large datasets is often hampered by the presence of noisy labels, which can arise from automated data collection or human annotation errors. These noisy labels can degrade model performance. A key component in addressing this is understanding the "transition matrix" – how often a true label is flipped to a specific incorrect label. However, in real-world scenarios, this matrix is not only unknown but can also vary from one instance to another (instance-dependent). Estimating this instance-dependent transition matrix is challenging because it requires knowledge of the true labels, which are precisely what's missing. Existing methods struggle with this inherent difficulty, especially when dealing with complex noise patterns.

Methods

The proposed method, CoNL (Contrastive Label-Noise Learning), tackles this problem in two main stages:

  1. Contrastive Co-Selecting: This stage leverages the power of self-supervised learning, specifically using a Momentum Contrast (MoCo) framework. The idea is that SSL can learn robust visual representations that are less susceptible to label noise. By training a backbone network with MoCo, the authors obtain these representations. Then, two distinct classifiers and transition matrices are trained simultaneously using these representations and the noisy labels. Crucially, these classifiers are trained with different data augmentation strategies (strong and weak) to encourage diversity in their learning abilities. This allows them to identify and select "confident" examples that are likely to have correct or near-correct labels, effectively filtering out some of the noise.
  2. Constraint T(x) Revision: In this stage, the selected confident examples are used to refine the estimated instance-dependent transition matrix. The core principle is that a good transition matrix should minimize classification losses on both the noisy labels and the inferred clean labels of these confident examples. By minimizing these losses, the transition matrix is iteratively improved. This process also allows for fine-tuning of the classifier head and the backbone model, further enhancing overall performance.

Datasets

The authors evaluated their method on several datasets:

  • Synthetic Datasets:

    • Fashion-MNIST
    • SVHN
    • CIFAR-10 These datasets were corrupted with instance-dependent label noise at various rates (10% to 50%).
  • Real-world Dataset:

    • CIFAR-10N This dataset contains human-annotated noisy labels with five different noise types.
  • Additional Datasets for Robustness:

    • CIFAR-100
    • WebVision These were used to test the method's performance with a larger number of classes and in a more challenging semi-supervised setting.

Key Insights and Ideas

  • SSL as a Noise Filter: The paper demonstrates that self-supervised learning, particularly contrastive learning, is not just for representation learning but can be a powerful tool for inferring cleaner labels in the presence of noise. The learned representations are inherently more robust to label errors.
  • Diversity in Classifiers for Better Selection: Training two classifiers with different augmentation strategies (strong vs. weak) is key. This diversity allows them to identify different types of confident examples, leading to a more comprehensive and reliable set of clean-label candidates.
  • Iterative Refinement of Transition Matrix: The "Constraint T(x) Revision" stage highlights the benefit of using the inferred clean labels to iteratively improve the transition matrix estimation. This feedback loop is crucial for achieving high accuracy, especially under high noise levels.
  • Instance-Dependent Noise is a Major Hurdle: The paper strongly emphasizes that accurately estimating instance-dependent transition matrices is significantly harder than estimating a global one, and CoNL's success in this area is a key contribution.

Key Results

CoNL achieved state-of-the-art performance across various datasets and noise levels. For instance:

  • On Fashion-MNIST with 50% instance-dependent noise, CoNL reached an accuracy of 86.01%, significantly outperforming other methods.
  • On SVHN with 50% noise, CoNL achieved 86.51% accuracy.
  • On CIFAR-10 with 50% noise, CoNL reached 84.79% accuracy.
  • The paper also shows that the "Constraint T(x) Revision" stage effectively reduces the estimation error of the transition matrix.

Limitations and Future Work

  • Hyperparameter Sensitivity: While not explicitly detailed as a limitation, the method relies on hyperparameters like λ and τ for confident example selection, which might require careful tuning.
  • Computational Cost: The use of MoCo for representation learning and the iterative refinement process likely increase the computational cost compared to simpler methods.
  • Future Work: The paper suggests that the proposed method can be extended to other semi-supervised learning tasks where label noise is present. Further exploration into more advanced SSL techniques and more sophisticated ways to leverage the learned representations for transition matrix estimation could be beneficial.

Summary updated

Abstract

The transition matrix reveals the transition relationship between clean labels and noisy labels. It plays an important role in building statistically consistent classifiers. In real-world applications, the transition matrix is usually unknown and has to be estimated. It is a challenging task to accurately estimate the transition matrix, especially when it depends on the instance. Given that both instances and noisy labels are available, the major difficulty of learning the transition matrix comes from the missing of clean information. A lot of methods have been proposed to infer clean information. The self-supervised learning has demonstrated great success. These methods could even achieve comparable performance with supervised learning on some datasets but without requiring any labels during the training. It implies that these methods can efficiently infer clean labels. Motivated by this, in this paper, we have proposed a practical method that leverages self-supervised learning to help learn the instance-dependent transition matrix. Empirically, the proposed method has achieved state-of-the-art performance on different datasets.
BibTeX
@inproceedings{lin2023improving,
  author = {Yexiong Lin and Yu Yao and Zhaoqing Wang and Xu Shen and Jun Yu and Bo Han and Tongliang Liu},
  title = {{Improving the Estimation of Instance-dependent Transition Matrix by using Self-supervised Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=d_w12b7fb20},
}
Figure