Understanding Generalization of Deep Neural Networks Trained with Noisy Labels

Wei Hu, Zhiyuan Li, Dingli Yu

Abstract: Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data. Such over-fitting ability hinders generalization when mislabeled training examples are present. On the other hand, simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios, a phenomenon not theoretically understood. This paper proposes and analyzes two simple and intuitive regularization methods: (i) regularization by the distance between the network parameters to initialization, and (ii) adding a trainable auxiliary variable to the network output for each training example. Theoretically, we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels. Our generalization analysis relies on the connection between wide neural network and neural tangent kernel (NTK). The generalization bound is independent of the network size, and is comparable to the bound one can get when there is no label noise. Experimental results verify the effectiveness of these methods on noisily labeled datasets.

Introduction

Modern deep neural networks are massively over-parameterized, giving them the capacity to memorize entire datasets—even if the labels are completely random. While this capacity is beneficial for clean data, real-world datasets are notoriously messy and often contain mislabeled examples. If a standard neural network is trained on noisy data for too long, it will eventually memorize the incorrect labels, causing its performance on unseen, clean data to plummet.

A common practical heuristic to prevent this is early stopping—halting training before the model starts memorizing the noise. However, early stopping is theoretically poorly understood. This research aims to bridge this gap by proposing simple, mathematically grounded regularization methods that prevent overfitting to noisy labels while guaranteeing good generalization on the underlying clean data.

Proposed Regularization Methods

The authors introduce two intuitive regularization techniques that require minimal changes to standard training pipelines:

  1. Regularization by Distance to Initialization (RDI): Instead of standard weight decay (which penalizes absolute weight magnitude), RDI penalizes the $L_2$ distance of the network's weights $\theta$ from their initial random values $\theta(0)$. The loss penalty is $\lambda \|\theta - \theta(0)\|^2$.
  2. Auxiliary Variables (AUX): For every training example $i$, a trainable scalar variable $b_i$ is added to the network's output. The network attempts to fit the label using $f(\theta, x_i) + \lambda b_i$. The idea is that the network learns the "easy" clean patterns, while the $b_i$ variables act as slack to absorb the "hard" noisy labels. At test time, the auxiliary variables are discarded.

Theoretically, these methods are analyzed through the lens of the Neural Tangent Kernel (NTK). In the limit of infinite width, both RDI and AUX are mathematically equivalent to Kernel Ridge Regression using the NTK.

Key Insights and Theoretical Analysis

The most elegant theoretical insight is that in wide networks, both RDI and AUX converge to the same training dynamics. They both act as Kernel Ridge Regression, a well-understood method for preventing overfitting. The authors mathematically prove that training with these regularizers on noisy data yields a generalization bound on the clean data distribution comparable to the bound obtained from perfectly clean labels.

Training and test error curves showing how AUX and RDI prevent overfitting compared to vanilla SGD.

Furthermore, these regularizers lead to monotonic validation curves. In standard training with noisy labels, validation accuracy usually peaks early and then crashes. With the proposed regularizers, validation performance remains stable and increases monotonically, eliminating the need for precise early stopping.

Experimental Results

The methods were evaluated on standard image classification benchmarks (MNIST and CIFAR-10) with artificially injected label noise.

  • Performance vs. Early Stopping: On binary MNIST and CIFAR tasks, both RDI and AUX consistently outperform vanilla Gradient Descent (GD) and achieve test errors on par with perfectly timed early stopping.
  • Scalability to Deep Networks: When applied to a ResNet-34 on CIFAR-10 with 40% label noise, standard training with early stopping achieves ~86.35% accuracy. Adding the AUX method boosts this to 88.95%, outperforming specialized robust loss functions.
Accuracy of ResNet-34 on CIFAR-10 with 40% label noise, comparing standard losses with and without the AUX method.

At an extreme 60% noise level, AUX achieves ~83.95% compared to standard early stopping's 79.13%. Empirical tracking also confirms that weights in regularized networks stay much closer to their initialization, validating the core intuition behind RDI.

Limitations and Future Work

The theoretical guarantees strictly rely on the network operating in the NTK regime. However, practical architectures like ResNet-34 likely operate outside this strict regime. Explaining why these methods remain effective in practice is an open theoretical question. Additionally, both methods introduce a regularization strength hyperparameter ($\lambda$) that may require tuning in real-world scenarios where the exact noise level is unknown.

Summary updated

Abstract

Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data. Such over-fitting ability hinders generalization when mislabeled training examples are present. On the other hand, simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios, a phenomenon not theoretically understood. This paper proposes and analyzes two simple and intuitive regularization methods: (i) regularization by the distance between the network parameters to initialization, and (ii) adding a trainable auxiliary variable to the network output for each training example. Theoretically, we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels. Our generalization analysis relies on the connection between wide neural network and neural tangent kernel (NTK). The generalization bound is independent of the network size, and is comparable to the bound one can get when there is no label noise. Experimental results verify the effectiveness of these methods on noisily labeled datasets.
BibTeX
@inproceedings{hu2020understanding,
  author = {Wei Hu and Zhiyuan Li and Dingli Yu},
  title = {{Understanding Generalization of Deep Neural Networks Trained with Noisy Labels}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=c31KSjnZFC0},
}
Figure