Abstract
Supervised learning typically assumes clean labels, but real-world datasets often contain significant label noise. While common practice involves designing complex robust loss functions to prevent overfitting, this paper demonstrates that the standard accuracy metric is theoretically robust under class-conditional label noise. We prove that maximizing accuracy on a noisy distribution is equivalent to maximizing it on the clean one. Leveraging this insight, we show that noisy validation sets are reliable for model selection and introduce the Noisy best Teacher and Student (NTS) framework. Our approach achieves state-of-the-art results on benchmark and real-world datasets like Clothing1M without requiring any clean data.
Introduction
In the world of supervised learning, labels are often treated as ground truth. However, datasets sourced from crowdsourcing or web-scraping are frequently "noisy." The standard wisdom suggests that directly maximizing accuracy on such data leads to overfitting, necessitating complex "robust" loss functions. This research challenges that notion, proving that the humble accuracy metric is far more resilient than previously believed.
Learning with noisy labels involves two primary hurdles:
- Training: Preventing the model from "memorizing" incorrect labels.
- Validation: Tuning hyperparameters or performing early stopping without a perfectly clean validation set.
While most research focuses on the first problem, the second is often bypassed by assuming access to a small "gold standard" clean dataset—a luxury not always available. This paper asks: Can we just trust the accuracy metric on the noisy data we already have?
Theoretical Framework
The authors establish three key theoretical pillars to support the use of accuracy in noisy environments:
- Theorem 1 (Accuracy Robustness): Under "diagonally-dominant" noise (where the correct label is still more likely than any specific incorrect one), a classifier that maximizes accuracy on the noisy distribution also maximizes it on the clean distribution.
- Theorem 2 (Generalization Bound): Using VC-dimension-based bounds, the authors show that with enough noisy samples, training accuracy converges toward $1 - \epsilon$, where $\epsilon$ is the noise rate.
- Theorem 3 (Validation Bound): Accuracy on a noisy validation set is proved to be a reliable, unbiased estimator of the model's performance on the underlying noisy distribution.
The NTS Framework
Based on these theoretical insights, the authors propose the NTS (Noisy best Teacher and Student) framework. This approach avoids the need for clean data entirely by following a two-stage process:
- Noisy best Teacher (NT): A model is trained on noisy data, and a noisy validation set is used to select the best epoch or version.
- Noisy best Student (NS): The teacher's predictions serve as "pseudo-labels" to train a student model. Again, the noisy validation set is used for model selection.
Experimental Evaluation
The researchers evaluated their findings on several benchmarks, including CIFAR-10 and CIFAR-100 with synthetic noise, and the large-scale Clothing1M dataset, which contains 1 million images with natural web-label noise.
Key Insights
- The $1 - \epsilon$ Limit: While a Deep Neural Network can reach 100% accuracy on a finite noisy training set via memorization, the theoretical maximum on the noisy distribution is $1 - \epsilon$. As data volume increases, the model is forced to prioritize clean patterns to reach this limit.
- Data Augmentation: Theoretical analysis suggests that data augmentation narrows the gap between finite samples and the true noisy distribution, aiding convergence to the optimal classifier.
- Noisy Validation is Valid: Practically, a noisy validation set is statistically sufficient to distinguish good models from bad ones, provided the noise is class-conditional.
Results
On the Clothing1M dataset, using a noisy validation set for model selection yielded results nearly identical to using a clean one. The NTS framework consistently outperformed standard Cross-Entropy (CE) and other robust methods like Co-teaching. For instance, on CIFAR-10 with 60% uniform noise, NTS improved CE accuracy from approximately 59% to 87%.
Conclusion and Future Work
The accuracy metric is robust enough to serve as a reliable guide even in the presence of significant label noise. This finding simplifies the machine learning pipeline by validating the use of noisy sets for model selection.
However, limitations remain. The current proofs rely on class-conditional noise and may not fully extend to "instance-dependent" noise, where specific images are inherently harder to label. Furthermore, the NTS approach relies on the teacher's initial accuracy; future work could explore iterative refinement to further mitigate potential biases.
Summary updated