PNP: Robust Learning from Noisy Labels by Probabilistic Noise Prediction

Zeren Sun, Fumin Shen, Dan Huang, Qiong Wang, Xiangbo Shu, Yazhou Yao, Jinhui Tang

Introduction

Deep Neural Networks (DNNs) are highly effective at learning from data but are notoriously susceptible to the "memorization effect," where they eventually fit even incorrect training labels. This leads to poor generalization, especially in real-world datasets where labels are often noisy due to automated collection methods like web scraping.

Existing solutions typically rely on sample selection, attempting to identify "clean" samples based on low loss values. However, these methods suffer from two primary drawbacks: high sensitivity to manually-tuned selection thresholds and a failure to distinguish between in-distribution (ID) noise (incorrect labels within the dataset's classes) and out-of-distribution (OOD) noise (samples that do not belong to any target class). We propose PNP (Probabilistic Noise Prediction), a framework that explicitly models these noise types probabilistically.

Methodology

PNP moves away from brittle hard-filtering by treating noise detection as a continuous prediction task. The system utilizes a dual-network architecture with a shared feature extraction backbone.

The PNP architecture: A shared backbone feeds into a Label Predictor Network (LPN) for classification and a Noise Predictor Network (NPN) to identify Clean, ID Noisy, and OOD Noisy samples.

Dual-Network Architecture

The architecture consists of two parallel heads:

  1. Label Predictor Network (LPN): Responsible for predicting the object category.
  2. Noise Predictor Network (NPN): Predicts a 3-way probability distribution—Clean, ID Noisy, or OOD Noisy.

Probabilistic Modeling and Loss Functions

In the PNP-soft variant, every sample contributes to training, but its influence is scaled by the NPN's confidence in its noise type. This "soft selection" acts as a safety net against incorrect noise predictions.

  • ID Noisy Samples: These are treated with a "sharpening" operation to encourage the model to identify the correct label within the known classes.
  • OOD Noisy Samples: These are "flattened," minimizing their impact on the classifier while still allowing the model to learn general visual features from them.
  • Consistency Regularization: The model enforces that different augmented views of the same sample produce consistent predictions, enhancing stability.

Experiments and Results

The authors evaluated PNP on both synthetic noise (CIFAR100N, CIFAR80N) and real-world web noise (Web-Aircraft, Web-Bird, Web-Car, and Food101N).

Comparison of PNP variants against state-of-the-art methods on CIFAR100N and CIFAR80N. PNP-soft and PNP-hard consistently outperform previous methods across various noise scenarios.

As shown in the experimental results, PNP consistently outperforms previous state-of-the-art (SOTA) methods like Jo-SRC and DivideMix. On the CIFAR80N dataset with 20% symmetric noise, PNP-soft achieved an accuracy of 67.00%, a significant improvement over previous benchmarks. In extreme noise scenarios (e.g., 80% symmetric noise), PNP maintains a substantial lead. On real-world data such as the Web-Car dataset, PNP-soft reached 90.11% accuracy, demonstrating its effectiveness in "in-the-wild" conditions.

Conclusion

PNP provides a robust framework for learning from noisy labels by explicitly and probabilistically distinguishing between clean, ID noisy, and OOD noisy samples. By replacing hard thresholds with soft, probabilistic weighting, PNP achieves superior performance and stability across a wide range of noise types and intensities. Future work will explore integrating self-supervised pre-training to further enhance robustness during the initial training phases.

Summary updated

Abstract

Label noise has been a practical challenge in deep learning due to the strong capability of deep neural networks in fitting all training data. Prior literature primarily resorts to sample selection methods for combating noisy labels. However, these approaches focus on dividing samples by order sorting or threshold selection, inevitably introducing hyperparameters (e.g., selection ratio / threshold) that are hard-to-tune and dataset-dependent. To this end, we propose a simple yet effective approach named PNP (Probabilistic Noise Prediction) to explicitly model label noise. Specifically, we simultaneously train two networks, in which one predicts the category label and the other predicts the noise type. By predicting label noise probabilistically, we identify noisy samples and adopt dedicated optimization objectives accordingly. Finally, we establish a joint loss for network update by unifying the classification loss, the auxiliary constraint loss, and the in-distribution consistency loss. Comprehensive experimental results on synthetic and real-world datasets demonstrate the superiority of our proposed method.
BibTeX
@inproceedings{sun2022pnp,
  author = {Zeren Sun and Fumin Shen and Dan Huang and Qiong Wang and Xiangbo Shu and Yazhou Yao and Jinhui Tang},
  title = {{PNP: Robust Learning from Noisy Labels by Probabilistic Noise Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00524},
}
Figure