Is your noise correction noisy? PLS: Robustness to label noise with two stage detection

Paul Albert, Eric Arazo, Tarun Krishna, Noel E. O'Connor, Kevin McGuinness

Abstract

Training neural networks on large-scale, web-crawled data often necessitates "correcting" noisy labels using model-generated guesses, known as pseudo-labels. However, if these guesses are erroneous, the model can fall into a feedback loop of errors called confirmation bias. This paper introduces Pseudo-Loss Selection (PLS), a two-stage framework that utilizes a novel "pseudo-loss" metric to filter out incorrect label corrections. By distinguishing between reliable and unreliable pseudo-labels, PLS significantly enhances robustness against both in-distribution and out-of-distribution noise.

Introduction: The Challenge of Uncurated Data

Deep learning's success is deeply tied to massive datasets, yet high-quality human annotation remains prohibitively expensive. Consequently, researchers frequently rely on "uncurated" data scraped from the web. Such data is inherently "noisy," containing labels that are either incorrect (in-distribution noise) or belong to categories outside the target classes (out-of-distribution noise).

Current state-of-the-art methods typically address this by detecting noisy samples and then attempting to "guess" their true labels via semi-supervised learning. The critical flaw in this approach is that these guesses are not always accurate. When a model trains on its own incorrect guesses, it develops confirmation bias, becoming increasingly confident in its mistakes and ultimately degrading overall performance. The authors of this paper pose a fundamental question: Is your noise correction itself noisy?

Methodology: Pseudo-Loss Selection (PLS)

The authors propose a two-stage detection pipeline designed to mitigate the impact of incorrect pseudo-labels.

Conceptual comparison between standard label noise robust algorithms and the proposed PLS framework. PLS adds a filtering stage to prevent incorrect label guesses from poisoning the training process.
  1. Noise Detection: Initially, noisy samples are identified by fitting a Gaussian Mixture Model (GMM) to the training loss. Noisy samples typically exhibit higher losses during the early stages of training.
  2. Pseudo-Labeling & Filtering: For samples flagged as noisy, the model generates a "pseudo-label." The core innovation is the introduction of pseudo-loss, defined as the cross-entropy between the model’s current guess and its prediction on an unaugmented version of the image.
  3. The Second GMM: The authors discovered that this pseudo-loss is bi-modal, effectively separating correct guesses from incorrect ones. By fitting a second GMM to the pseudo-loss, the framework distinguishes between "confidently corrected" samples and "under-confident" ones.
  4. Confidence-Guided Contrastive Learning: Rather than discarding under-confident samples, PLS employs a hybrid objective. Confident samples are trained with standard supervised loss, while under-confident samples are utilized in a contrastive learning objective. This allows the model to learn meaningful visual representations even when the specific label remains uncertain.

Key Insights: The Power of Bi-modality

The most significant technical insight is the bi-modality of pseudo-loss. As training progresses, the distribution of pseudo-loss for correct and incorrect corrections becomes increasingly distinct, allowing for precise filtering.

The distribution of pseudo-loss over training epochs. The bi-modal nature allows for effective separation of correct and incorrect label corrections, reducing confirmation bias.
  • Incorrect Corrections are Poisonous: The research demonstrates that training on an incorrect pseudo-label is often more detrimental than treating the image as unlabeled data.
  • Handling OOD Noise: Most robust algorithms struggle when an image does not belong to any target class. By applying a contrastive objective to under-confident samples, PLS extracts value from OOD images without forcing them into incorrect categories.
  • Dynamic Weighting: Unlike hard-thresholding methods (e.g., keeping labels with >0.95 confidence), PLS uses GMM posterior probabilities to softly down-weight samples, allowing for better adaptation throughout the training process.

Datasets and Evaluation

PLS was evaluated on a comprehensive suite of benchmarks to test its versatility:

  • CIFAR-100: Synthetically corrupted with symmetric and asymmetric in-distribution (ID) noise.
  • miniImageNet: Corrupted with both ID and out-of-distribution (OOD) noise (e.g., images from Places365).
  • Controlled Web Noise (CNWL): Designed to simulate real-world web crawl conditions.
  • Real-world Web Datasets: Fine-grained classification tasks including Web-Aircraft, Web-bird, and Web-car.

Results and Performance

PLS consistently outperformed existing state-of-the-art methods, such as DivideMix and JoSRC, across various noise levels and types:

  • Extreme Noise: On CIFAR-100 with an 80% noise rate, PLS achieved 63.33% accuracy, significantly outperforming competitors.
  • OOD Robustness: In scenarios with heavy OOD noise (e.g., 60% on miniImageNet), PLS maintained high performance while other methods saw sharp declines.
  • Real-World Benchmarks: On the Web-Aircraft dataset, PLS reached a new benchmark accuracy of 87.58%.

Conclusion and Future Directions

While PLS represents a significant advancement, the authors identify areas for future work. The method's effectiveness depends on a "warmup" phase; if noise is excessively high during this period, initial detection may be compromised. Additionally, the two-stage GMM and contrastive learning branch increase computational overhead, though the accuracy gains often justify this cost. Finally, while PLS improves correction, the initial detection of complex web noise remains a challenging area for further research.

Summary updated

Abstract

Designing robust algorithms capable of training accurate neural networks on uncurated datasets from the web has been the subject of much research as it reduces the need for time consuming human labor. The focus of many previous research contributions has been on the detection of different types of label noise; however, this paper proposes to improve the correction accuracy of noisy samples once they have been detected. In many state-of-the-art contributions, a two phase approach is adopted where the noisy samples are detected before guessing a corrected pseudo-label in a semi-supervised fashion. The guessed pseudo-labels are then used in the supervised objective without ensuring that the label guess is likely to be correct. This can lead to confirmation bias, which reduces the noise robustness. Here we propose the pseudo-loss, a simple metric that we find to be strongly correlated with pseudo-label correctness on noisy samples. Using the pseudo-loss, we dynamically down weight under-confident pseudo-labels throughout training to avoid confirmation bias and improve the network accuracy. We additionally propose to use a confidence guided contrastive objective that learns robust representation on an interpolated objective between class bound (supervised) for confidently corrected samples and unsupervised representation for under-confident label corrections. Experiments demonstrate the state-of-the-art performance of our Pseudo-Loss Selection (PLS) algorithm on a variety of benchmark datasets including curated data synthetically corrupted with in-distribution and out-of-distribution noise, and two real world web noise datasets. Our experiments are fully reproducible github.com/PaulAlbert31/SNCF
BibTeX
@inproceedings{albert2023is,
  author = {Paul Albert and Eric Arazo and Tarun Krishna and Noel E. O'Connor and Kevin McGuinness},
  title = {{Is your noise correction noisy? PLS: Robustness to label noise with two stage detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00020},
}
Figure