Skeptical Deep Learning with Distribution Correction

Mingxiao An, Yongzhou Chen, Qi Liu, Chuanren Liu, Guangyi Lv, Fangzhao Wu, Jianhui MaarXiv 2018 · OpenReview

Introduction

Deep neural networks (DNNs) have achieved remarkable success across various domains, largely driven by the availability of massive datasets. However, the cost and difficulty of obtaining perfectly labeled data at scale often lead to the use of "noisy" labels—labels that are incorrect due to human error, crowdsourcing inconsistencies, or even adversarial attacks.

The core challenge lies in the inherent nature of DNNs: they are exceptional memorizers. Given enough capacity, a model will eventually overfit even random noise. Standard loss functions, such as Cross-Entropy (Log Loss), exacerbate this issue by penalizing "incorrect" predictions with infinite loss as the predicted probability of the given label approaches zero. This forces the model to "chase" noisy outliers, degrading its generalization performance on clean data.

The Distribution Correction Framework

To address this, the authors introduce a general framework for Distribution Correction. This approach moves beyond simple loss adjustment by mathematically modeling how the expectation of a "clean" (correct) distribution can be recovered from an "incorrect" (noisy) distribution.

Conceptual framework for distribution correction, mapping noisy datasets back to the "reality" of clean distributions.

As illustrated in the framework above, the goal is to implement a correction mechanism—often as weights on the gradient—that allows the model to learn from the "distorted reality" of a noisy dataset as if it were sampling from the "true reality" of a clean dataset. The authors demonstrate that existing techniques, like Forward Loss Correction, are specific instances of their broader Conditional Correction framework.

Skeptical Learning: Loss and Adaptation

The practical implementation of this framework relies on two key components: the Skeptical Loss and an Adaptive Transition Matrix.

1. Skeptical Loss ($L_{SK}$)

Unlike the standard Log Loss, the proposed Skeptical Loss is a bounded function. By capping the maximum penalty a single sample can incur, the model becomes "skeptical" of samples where its own prediction strongly disagrees with the provided label. This prevents the "gradient explosion" typically caused by noisy outliers, allowing the training process to focus on the underlying patterns shared by the majority of correctly labeled data.

2. Adaptive Transition Matrix ($\hat{T}$)

The model maintains and updates an online estimate of the probability that a true label $y$ has been flipped to a noisy label $\tilde{y}$. This matrix, $\hat{T}$, is updated based on the model's own confident predictions. If the model is highly certain ($p > 1 - \epsilon$) about a class that contradicts the provided label, it updates the matrix to reflect that this specific label flip is likely a result of noise. This allows the model to effectively "clean" the dataset on the fly without requiring prior knowledge of the noise distribution.

Benchmarking with Confusing Label Noise

A significant contribution of this work is the introduction of a more realistic noise model: Confusing Label Noise. While most research evaluates robustness using "Symmetric Noise" (random flipping), real-world errors are often feature-dependent—for instance, a "cat" is far more likely to be mislabeled as a "lynx" than as a "car." The authors developed an algorithm using a teacher model to generate these high-quality, confusing noisy labels, creating a more rigorous testbed for robust learning.

Experimental Results

The "Skeptical" approach was evaluated on standard benchmarks including Fashion-MNIST, CIFAR-10, and ImageNet. Across all datasets, the method consistently outperformed standard Log Loss, Unhinged Loss, and static correction methods.

Performance comparison across varying label noise rates, demonstrating the robustness of the conditional skeptical approach.

As shown in the results above, the performance gap between the Skeptical method (red line) and standard approaches widens significantly as the label noise rate increases. For example, on CIFAR-10 with 40% confusing noise, the Skeptical method achieved a test error of 35.24%, a substantial improvement over the 41.65% error rate of standard Log Loss. Furthermore, the model demonstrated high precision in label recovery, often identifying correct labels with 80-85% accuracy even in high-noise environments.

Conclusion and Future Outlook

By integrating a bounded loss function with an adaptive mechanism for distribution correction, "Skeptical Learning" provides a robust path forward for training DNNs on imperfect data. While the method requires a "warm-up" period to build initial confidence and involves some hyperparameter tuning, its ability to distinguish truth from error without prior noise information is a significant step toward more resilient AI systems. Future research may extend these principles to domains like Natural Language Processing or Reinforcement Learning, where label and reward noise remain persistent challenges.

Summary updated

Abstract

Recently deep neural networks have been successfully used for various classification tasks, especially for problems with massive perfectly labeled training data. However, it is often costly to have large-scale credible labels in real-world applications. One solution is to make supervised learning robust with imperfectly labeled input. In this paper, we develop a distribution correction approach that allows deep neural networks to avoid overfitting imperfect training data. Specifically, we treat the noisy input as samples from an incorrect distribution, which will be automatically corrected during our training process. We test our approach on several classification datasets with elaborately generated noisy labels. The results show significantly higher prediction and recovery accuracy with our approach compared to alternative methods.
BibTeX
@misc{an2018skeptical,
  author = {Mingxiao An and Yongzhou Chen and Qi Liu and Chuanren Liu and Guangyi Lv and Fangzhao Wu and Jianhui Ma},
  title = {{Skeptical Deep Learning with Distribution Correction}},
  year = {2018},
  eprint = {1811.03821},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.03821},
}
Figure