Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels

Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, Pheng-Ann Heng

Abstract

Standard deep learning training often struggles with noisy labels, as models tend to memorize errors. This paper provides a theoretical justification for using standard accuracy and noisy validation sets. We prove that maximizing accuracy on a noisy distribution is mathematically guaranteed to yield an optimal classifier for the clean distribution, provided sufficient data is available. We introduce the Noisy best Teacher and Student (NTS) framework, which leverages these insights to achieve state-of-the-art performance on corrupted datasets without requiring clean validation data.

Introduction

In the era of big data, labels are often sourced from imperfect channels like web scraping or crowdsourcing. This leads to label noise, which conventional wisdom suggests will cause deep neural networks to overfit and fail. Consequently, researchers have focused on complex robust loss functions or the acquisition of small, high-quality "clean" validation sets. This work challenges those assumptions by examining the inherent robustness of the accuracy metric itself and providing a theoretical "green light" for standard training practices under specific conditions.

Theoretical Foundations

The authors establish three key theorems to support the use of noisy data for both training and model selection.

  1. Metric Robustness (Theorem 1): Under class-conditional noise, a classifier that maximizes accuracy on the noisy distribution also maximizes it on the clean distribution, provided the noise is "diagonally dominant" (the correct label remains the most probable).
  2. Generalization Bound (Theorem 2): Using VC dimension theory, the authors show that the gap between empirical training accuracy and true noisy distribution accuracy vanishes as the number of noisy samples increases.
  3. Validation Reliability (Theorem 3): A noisy validation set is proven to be a reliable proxy for model selection, justifying its use for hyperparameter tuning and early stopping.
Overview of the theoretical framework and NTS pipeline.

The NTS Framework

Building on these theorems, the Noisy best Teacher and Student (NTS) framework is proposed. The process consists of two primary steps:

  • Step 1 (Teacher): Train a model on noisy data and use a noisy validation set to pick the best-performing checkpoint. This model serves as the "Teacher."
  • Step 2 (Student): Use the Teacher’s predictions to "clean" the labels (pseudo-labeling) and train a "Student" model from scratch on this refined dataset.

The Three Phases of Learning

A pivotal insight of this work is the identification of three distinct phases in learning with noise, governed by the sample size ($m$):

  • Phase 1 (Small $m$): The model suffers from high variance and underfitting.
  • Phase 2 (Medium $m$): The model has enough capacity to memorize the noise perfectly, leading to high training accuracy but poor clean test accuracy.
  • Phase 3 (Large $m$): With a sufficiently large dataset, the underlying signal of the clean distribution becomes dominant. The model converges toward the global optimum for the clean data, effectively overcoming the noise.
Impact of sample size on model convergence and decision boundaries.

Experimental Results

The framework was evaluated on CIFAR-10/100 and the real-world Clothing1M dataset.

  • Clothing1M: Using a noisy validation set to select the teacher achieved 70.21% accuracy, which was nearly identical to (and sometimes better than) using a clean validation set (70.02%).
  • CIFAR-10: With 60% uniform noise, the NTS student reached 87.88% accuracy, a massive jump from the baseline's 59.15%.

Conclusion and Limitations

This research demonstrates that standard accuracy is a robust metric for learning with noisy labels. While the theory currently assumes diagonally-dominant class-conditional noise, the practical results on Clothing1M suggest broad applicability. Future work may extend these proofs to instance-dependent noise, where error rates vary based on specific image features.

Summary updated

BibTeX
@inproceedings{chen2021robustness,
  author = {Pengfei Chen and Junjie Ye and Guangyong Chen and Jingwei Zhao and Pheng-Ann Heng},
  title = {{Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i13.17364},
}
Figure