Introduction
Deep Neural Networks (DNNs) are highly effective at learning from data but are notoriously susceptible to the "memorization effect," where they eventually fit even incorrect training labels. This leads to poor generalization, especially in real-world datasets where labels are often noisy due to automated collection methods like web scraping.
Existing solutions typically rely on sample selection, attempting to identify "clean" samples based on low loss values. However, these methods suffer from two primary drawbacks: high sensitivity to manually-tuned selection thresholds and a failure to distinguish between in-distribution (ID) noise (incorrect labels within the dataset's classes) and out-of-distribution (OOD) noise (samples that do not belong to any target class). We propose PNP (Probabilistic Noise Prediction), a framework that explicitly models these noise types probabilistically.
Methodology
PNP moves away from brittle hard-filtering by treating noise detection as a continuous prediction task. The system utilizes a dual-network architecture with a shared feature extraction backbone.
Dual-Network Architecture
The architecture consists of two parallel heads:
- Label Predictor Network (LPN): Responsible for predicting the object category.
- Noise Predictor Network (NPN): Predicts a 3-way probability distribution—Clean, ID Noisy, or OOD Noisy.
Probabilistic Modeling and Loss Functions
In the PNP-soft variant, every sample contributes to training, but its influence is scaled by the NPN's confidence in its noise type. This "soft selection" acts as a safety net against incorrect noise predictions.
- ID Noisy Samples: These are treated with a "sharpening" operation to encourage the model to identify the correct label within the known classes.
- OOD Noisy Samples: These are "flattened," minimizing their impact on the classifier while still allowing the model to learn general visual features from them.
- Consistency Regularization: The model enforces that different augmented views of the same sample produce consistent predictions, enhancing stability.
Experiments and Results
The authors evaluated PNP on both synthetic noise (CIFAR100N, CIFAR80N) and real-world web noise (Web-Aircraft, Web-Bird, Web-Car, and Food101N).
As shown in the experimental results, PNP consistently outperforms previous state-of-the-art (SOTA) methods like Jo-SRC and DivideMix. On the CIFAR80N dataset with 20% symmetric noise, PNP-soft achieved an accuracy of 67.00%, a significant improvement over previous benchmarks. In extreme noise scenarios (e.g., 80% symmetric noise), PNP maintains a substantial lead. On real-world data such as the Web-Car dataset, PNP-soft reached 90.11% accuracy, demonstrating its effectiveness in "in-the-wild" conditions.
Conclusion
PNP provides a robust framework for learning from noisy labels by explicitly and probabilistically distinguishing between clean, ID noisy, and OOD noisy samples. By replacing hard thresholds with soft, probabilistic weighting, PNP achieves superior performance and stability across a wide range of noise types and intensities. Future work will explore integrating self-supervised pre-training to further enhance robustness during the initial training phases.
Summary updated