Probabilistic End-To-End Noise Correction for Learning With Noisy Labels.

Kun Yi, Jianxin WuCVPR 2019 · OpenReview

Abstract

Deep learning has achieved excellent performance in various computer vision tasks, but requires a lot of training examples with clean labels. It is easy to collect a dataset with noisy labels, but such noise makes networks overfit seriously and accuracies drop dramatically. To address this problem, we propose an end-to-end framework called PENCIL, which can update both network parameters and label estimations as label distributions. PENCIL is independent of the backbone network structure and does not need an auxiliary clean dataset or prior information about noise, thus it is more general and robust than existing methods and is easy to apply. PENCIL outperformed previous state-of-the-art methods by large margins on both synthetic and real-world datasets with different noise types and noise rates. Experiments show that PENCIL is robust on clean datasets, too.

The Challenge of Noisy Labels

Deep neural networks (DNNs) are notoriously good at memorizing data—including errors. While large-scale datasets like ImageNet are the bedrock of modern AI, collecting perfectly clean labels is expensive and slow. "Noisy" labels (incorrect annotations) are common in datasets scraped from the web or labeled by non-experts.

The challenge is that a DNN with enough capacity will eventually overfit to these noisy labels, causing its test accuracy to plummet after an initial peak. Existing solutions often require a "gold standard" subset of clean data or a pre-defined "noise transition matrix" (knowing exactly how likely a 'cat' is to be mislabeled as a 'dog'), neither of which is practical for real-world applications.

The PENCIL Framework

The authors introduce PENCIL (Probabilistic End-to-end Noise Correction In Labels). The core idea is to treat the label of each image as a learnable parameter, $y^d$, represented as a probability distribution. This allows the labels to "evolve" as the model becomes more confident about what an image actually contains.

The PENCIL framework architecture showing the interaction between network predictions, label distributions, and the three loss components.

As shown in the architecture above, the framework uses a specialized loss function composed of three parts:

  1. Classification Loss ($L_c$): A modified version of Kullback-Leibler (KL) divergence. By swapping the operands in the KL divergence formula, the authors created a mechanism where the model can "pull" the label toward the correct class during backpropagation.
  2. Compatibility Loss ($L_o$): This prevents the estimated label distributions from wandering too far from the original (albeit noisy) labels, acting as a regularizer.
  3. Entropy Loss ($L_e$): This encourages the network to eventually make "peaky" predictions (approaching a one-hot vector) rather than remaining uncertain.

Experimental Results and Insights

The authors evaluated PENCIL on major benchmarks including CIFAR-10/100, Clothing1M (1M+ real-world noisy images), and CUB-200.

One of the most significant findings is the framework's ability to actively correct labels during the training process. As illustrated in the figure below, the percentage of correct labels in the training set increases dramatically once the PENCIL learning phase begins.

Improvement in label accuracy over training epochs on CIFAR-10 with 30% asymmetric noise.

Key Performance Highlights:

  • Superiority in High Noise: On CIFAR-10 with 90% symmetric noise, PENCIL maintained 61.21% accuracy, while a standard Cross-Entropy approach collapsed to 16.30%.
  • Real-World SOTA: On the Clothing1M dataset, PENCIL achieved 73.49% accuracy, outperforming previous state-of-the-art methods without using any available clean training subsets.
  • Robustness on Clean Data: On the CUB-200 dataset (mostly clean), PENCIL actually improved accuracy slightly (82.64% vs 81.93%), suggesting it can even fix minor human errors present in "clean" datasets.

Conclusion

PENCIL represents a shift in perspective: treating labels not as fixed ground truth, but as variables to be optimized. This backbone-independent framework provides a robust, end-to-end solution for training high-performance models on imperfect data without requiring expensive auxiliary clean datasets.

Summary updated

Abstract

Deep learning has achieved excellent performance in various computer vision tasks, but requires a lot of training examples with clean labels. It is easy to collect a dataset with noisy labels, but such noise makes networks overfit seriously and accuracies drop dramatically. To address this problem, we propose an end-to-end framework called PENCIL, which can update both network parameters and label estimations as label distributions. PENCIL is independent of the backbone network structure and does not need an auxiliary clean dataset or prior information about noise, thus it is more general and robust than existing methods and is easy to apply. PENCIL outperformed previous state-of-the-art methods by large margins on both synthetic and real-world datasets with different noise types and noise rates. Experiments show that PENCIL is robust on clean datasets, too.
BibTeX
@inproceedings{yi2019probabilistic,
  author = {Kun Yi and Jianxin Wu},
  title = {{Probabilistic End-To-End Noise Correction for Learning With Noisy Labels.}},
  booktitle = {CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=rmENiJXl_aB},
}
Figure