Abstract
Training neural networks on large-scale, web-crawled data often necessitates "correcting" noisy labels using model-generated guesses, known as pseudo-labels. However, if these guesses are erroneous, the model can fall into a feedback loop of errors called confirmation bias. This paper introduces Pseudo-Loss Selection (PLS), a two-stage framework that utilizes a novel "pseudo-loss" metric to filter out incorrect label corrections. By distinguishing between reliable and unreliable pseudo-labels, PLS significantly enhances robustness against both in-distribution and out-of-distribution noise.
Introduction: The Challenge of Uncurated Data
Deep learning's success is deeply tied to massive datasets, yet high-quality human annotation remains prohibitively expensive. Consequently, researchers frequently rely on "uncurated" data scraped from the web. Such data is inherently "noisy," containing labels that are either incorrect (in-distribution noise) or belong to categories outside the target classes (out-of-distribution noise).
Current state-of-the-art methods typically address this by detecting noisy samples and then attempting to "guess" their true labels via semi-supervised learning. The critical flaw in this approach is that these guesses are not always accurate. When a model trains on its own incorrect guesses, it develops confirmation bias, becoming increasingly confident in its mistakes and ultimately degrading overall performance. The authors of this paper pose a fundamental question: Is your noise correction itself noisy?
Methodology: Pseudo-Loss Selection (PLS)
The authors propose a two-stage detection pipeline designed to mitigate the impact of incorrect pseudo-labels.
- Noise Detection: Initially, noisy samples are identified by fitting a Gaussian Mixture Model (GMM) to the training loss. Noisy samples typically exhibit higher losses during the early stages of training.
- Pseudo-Labeling & Filtering: For samples flagged as noisy, the model generates a "pseudo-label." The core innovation is the introduction of pseudo-loss, defined as the cross-entropy between the model’s current guess and its prediction on an unaugmented version of the image.
- The Second GMM: The authors discovered that this pseudo-loss is bi-modal, effectively separating correct guesses from incorrect ones. By fitting a second GMM to the pseudo-loss, the framework distinguishes between "confidently corrected" samples and "under-confident" ones.
- Confidence-Guided Contrastive Learning: Rather than discarding under-confident samples, PLS employs a hybrid objective. Confident samples are trained with standard supervised loss, while under-confident samples are utilized in a contrastive learning objective. This allows the model to learn meaningful visual representations even when the specific label remains uncertain.
Key Insights: The Power of Bi-modality
The most significant technical insight is the bi-modality of pseudo-loss. As training progresses, the distribution of pseudo-loss for correct and incorrect corrections becomes increasingly distinct, allowing for precise filtering.
- Incorrect Corrections are Poisonous: The research demonstrates that training on an incorrect pseudo-label is often more detrimental than treating the image as unlabeled data.
- Handling OOD Noise: Most robust algorithms struggle when an image does not belong to any target class. By applying a contrastive objective to under-confident samples, PLS extracts value from OOD images without forcing them into incorrect categories.
- Dynamic Weighting: Unlike hard-thresholding methods (e.g., keeping labels with >0.95 confidence), PLS uses GMM posterior probabilities to softly down-weight samples, allowing for better adaptation throughout the training process.
Datasets and Evaluation
PLS was evaluated on a comprehensive suite of benchmarks to test its versatility:
- CIFAR-100: Synthetically corrupted with symmetric and asymmetric in-distribution (ID) noise.
- miniImageNet: Corrupted with both ID and out-of-distribution (OOD) noise (e.g., images from Places365).
- Controlled Web Noise (CNWL): Designed to simulate real-world web crawl conditions.
- Real-world Web Datasets: Fine-grained classification tasks including Web-Aircraft, Web-bird, and Web-car.
Results and Performance
PLS consistently outperformed existing state-of-the-art methods, such as DivideMix and JoSRC, across various noise levels and types:
- Extreme Noise: On CIFAR-100 with an 80% noise rate, PLS achieved 63.33% accuracy, significantly outperforming competitors.
- OOD Robustness: In scenarios with heavy OOD noise (e.g., 60% on miniImageNet), PLS maintained high performance while other methods saw sharp declines.
- Real-World Benchmarks: On the Web-Aircraft dataset, PLS reached a new benchmark accuracy of 87.58%.
Conclusion and Future Directions
While PLS represents a significant advancement, the authors identify areas for future work. The method's effectiveness depends on a "warmup" phase; if noise is excessively high during this period, initial detection may be compromised. Additionally, the two-stage GMM and contrastive learning branch increase computational overhead, though the accuracy gains often justify this cost. Finally, while PLS improves correction, the initial detection of complex web noise remains a challenging area for further research.
Summary updated