A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels

Devraj Mandal, Shrisha Bharadwaj, Soma BiswasWACV 2020

TL;DR

This paper introduces mCT-S2R, a novel approach to train deep learning models effectively even when the training data is heavily corrupted with noisy labels. By combining self-supervised pre-training with a co-teaching strategy that intelligently re-labels uncertain samples, mCT-S2R achieves state-of-the-art performance on challenging noisy label benchmarks while being computationally more efficient.

What Problem Does This Paper Address?

Deep learning models have achieved remarkable success, largely due to the availability of massive, clean datasets. However, acquiring such datasets is prohibitively expensive. Consequently, many real-world applications rely on data sourced from the web or crowd-sourcing, which often contains noisy labels. These noisy labels can severely degrade the performance of deep neural networks, as they can even overfit to incorrect information. While various methods exist to combat label noise, they often struggle with high noise levels, require clean validation sets, or are computationally intensive. This paper aims to address the challenge of training robust deep learning models in the presence of significant label noise, without requiring any clean data for validation.

Methods

The proposed framework, mCT-S2R, builds upon the "Co-teaching" approach but introduces two key innovations: self-supervised pre-training and a novel re-labeling strategy.

  1. Self-Supervised Pre-training (RotNet): Before any label-dependent training, the network is pre-trained using a self-supervised task. Specifically, they employ the RotNet method, where the model learns to predict the rotation applied to an image (0, 90, 180, or 270 degrees). This allows the network to learn robust feature representations from the data distribution without relying on potentially noisy labels, mitigating overfitting. The initial convolutional layers of the subsequent classification networks are initialized with these pre-trained weights.
  2. Co-teaching with Small-Loss Instances: Two parallel networks are trained simultaneously, similar to Co-teaching. In each mini-batch, they identify "small-loss" instances – samples that the networks are more confident about. These small-loss instances are used to update the weights of both networks.
  3. Re-labeling Large-Loss Samples: This is a crucial step. Instead of discarding "large-loss" (potentially noisy) samples, the framework uses the features extracted from the small-loss samples to compute class-wise mean feature representations. Then, for each large-loss sample, its distance to these class means is calculated. A pseudo-label is assigned based on the closest mean, and a confidence score is derived from the softmax probability. Large-loss samples with high confidence scores are then selected to form a new dataset.
  4. Final Training with Augmented Data: The final classification model is trained using an augmented dataset comprising the original small-loss samples and the confidently re-labeled large-loss samples. This approach leverages potentially correct information from the noisy data, rather than discarding it. Notably, after the re-labeling phase, training proceeds with a single network, making it computationally lighter than traditional Co-teaching.

Datasets

The paper evaluates its approach on three standard benchmark datasets commonly used for noisy label research:

  • MNIST: A dataset of handwritten digits.
  • CIFAR-10: A dataset of 10 classes of everyday objects.
  • CIFAR-100: A dataset of 100 classes of objects, making it a more challenging task.

The authors introduce label noise by corrupting the original clean labels using two models: symmetry flipping and pair flipping, with varying noise levels (e.g., 45%, 50%, 20%).

Key Insights and Ideas

  • Self-supervision as a powerful pre-cursor: Pre-training with self-supervised tasks like RotNet is shown to be highly effective in learning robust features that are less susceptible to noisy labels, especially on more complex datasets like CIFAR-10/100.
  • Re-labeling is more than just discarding: Instead of simply ignoring uncertain samples, intelligently re-labeling them using learned feature representations can significantly boost performance by recovering valuable information from the noisy data.
  • Computational efficiency: By using two networks only for the initial small-loss identification and then switching to a single network for re-labeling and final training, the proposed method is more computationally efficient than methods that require two parallel networks throughout the entire training process.
  • Robustness to hyper-parameters: The framework demonstrates consistent performance across a range of hyper-parameter values (like Tupdate and κ), reducing the need for a dedicated clean validation set for tuning.

Key Results

The mCT-S2R framework achieves state-of-the-art results on the tested datasets under various noise conditions.

  • On MNIST, mCT-S2R achieves accuracies up to 99.45% (with pair-flip 45% noise) and 98.86% (with symmetry-20% noise), significantly outperforming existing methods.
  • For CIFAR-10, the performance improvement is even more pronounced, with mCT-S2R reaching up to 87.21% (pair-flip 45%) and 87.21% (symmetry-50%), demonstrating the effectiveness of self-supervision on more complex data.
  • On the challenging CIFAR-100 dataset, mCT-S2R achieves up to 60.49% (symmetry-20% noise), again showing substantial gains over other methods.
  • Ablation studies show that while self-supervision is less critical for the simpler MNIST dataset, it is crucial for the performance on CIFAR-10 and CIFAR-100. The re-labeling strategy also consistently improves results.

Limitations and Future Work

  • Dataset Dependency: While the method performs well, the paper notes that for the CIFAR-100 dataset, the pair-flip noise model (45%) can still be challenging, potentially due to the difficulty of correctly re-labeling in a dataset with 100 classes.
  • Hyper-parameter Tuning: Although the framework is shown to be robust to hyper-parameter choices, some tuning is still required. The paper suggests that the proposed model does not need an exclusive clean validation set for optimal tuning, but the choice of Tupdate and κ still influences performance.
  • Future Work: The authors suggest that further exploration into different self-supervised tasks and more advanced re-labeling strategies could lead to even greater improvements. The computational efficiency of the single-network final training phase also opens avenues for scaling to even larger datasets.

Summary updated

BibTeX
@inproceedings{mandal2020novel,
  author = {Devraj Mandal and Shrisha Bharadwaj and Soma Biswas},
  title = {{A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure