TL;DR
This paper provides a theoretical explanation for why the "small-loss criterion" works for training deep neural networks with noisy labels. They propose a refined version of this criterion that improves performance by considering class-specific losses and leveraging unlabeled data, demonstrating its effectiveness on benchmark datasets.
What Problem Does This Paper Address?
Deep neural networks (DNNs) excel when trained on large datasets with accurate labels. However, in real-world applications, labels are often collected through crowdsourcing or other less rigorous methods, leading to noisy labels. Blindly training DNNs on such data can severely degrade performance. While many methods have been developed to handle noisy labels, a theoretical understanding of why certain approaches, particularly those based on the "small-loss criterion," are effective has been lacking. This paper aims to bridge this gap by providing theoretical insights and a practical refinement of the small-loss criterion.
Methods
The core of this paper lies in its theoretical analysis. The authors first establish a condition (diagonally dominant noise transition matrix) under which learning the true concept from noisy labels is theoretically possible. They then prove that under this condition, examples with correct labels will have smaller losses than those with incorrect labels, even for a model that is close to the optimal one. This theoretically justifies the small-loss criterion.
Building on this, they propose a reformalized small-loss criterion (RSL). Key aspects of their approach include:
- Class-by-class selection: Instead of a global small-loss selection, they rank and select examples within each class to account for varying noise levels and the non-comparability of losses across classes.
- Mean loss over epochs: They use the average loss of an example over multiple training epochs to get a more stable signal than single-epoch loss, which can fluctuate significantly.
- Leveraging unlabeled data (RSL-WM): They integrate the selected (and potentially still noisy) data with the remaining unlabeled data using a semi-supervised learning method (MixMatch) to further improve performance.
- Weighted MixMatch: They re-weight the selected examples based on their mean loss to mitigate the impact of any remaining noise.
Datasets
The paper uses the following datasets for experiments:
- CIFAR-10 and CIFAR-100: Standard image classification datasets used for synthetic noise addition (uniform, pairwise, and structured noise).
- WebVision: A large-scale dataset with naturally occurring noisy labels, used to evaluate the method on real-world noisy data.
Key Insights and Ideas
- The "small-loss" criterion is theoretically grounded: The paper provides a formal proof that under a specific noise condition (diagonally dominant), examples with correct labels indeed have lower losses, explaining why this heuristic works.
- Class-specific noise matters: Losses are not directly comparable across different classes, especially with structured noise. Selecting small-loss examples on a per-class basis is crucial for effective noise handling.
- Mean loss is more reliable than single-epoch loss: Fluctuations in single-epoch losses make them unreliable for sample selection, whereas the cumulative mean loss provides a more stable and discriminative signal.
- Leveraging unlabeled data is beneficial: Even after selecting "small-loss" examples, the remaining data can still be valuable when treated as unlabeled data in a semi-supervised learning framework.
Key Results
- The proposed RSL and RSL-WM methods achieve state-of-the-art or competitive performance on CIFAR-10, CIFAR-100, and WebVision datasets across various noise types and levels.
- Experiments demonstrate that the diagonally-dominant condition is indeed important; when violated, the small-loss criterion can fail, with incorrect examples sometimes having lower losses than correct ones.
- The RSL-WM method, which leverages unlabeled data, consistently outperforms RSL (which only uses selected data), highlighting the benefit of semi-supervised learning.
- The paper shows that as the model
ggets closer to the theoretically optimal modelg*(simulated by using a larger fraction of the training data), the small-loss criterion becomes more effective, leading to higher precision in selected data.
Limitations and Future Work
- Noise rate estimation: While the proposed method is more robust to noise rate estimation errors than some others, it still relies on some assumptions or estimations of noise rates. The authors suggest that future work could explore extending the theoretical explanation to instance-dependent label noise, where noise is not uniform across all instances of a class.
- Hyperparameter tuning: The performance of RSL and RSL-WM depends on hyperparameters like β, γ, and κ. While default values are provided and sensitivity analyses are conducted, optimal tuning might still be required for specific datasets or noise patterns.
Summary updated