Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee

Wei Hu, Zhiyuan Li, Dingli Yu

TL;DR: This paper introduces two simple regularization techniques—Regularization by Distance to Initialization (RDI) and Auxiliary Variables (AUX)—that prevent over-parameterized neural networks from memorizing noisy labels. By leveraging the Neural Tangent Kernel (NTK) framework, the authors provide a theoretical guarantee that these methods generalize well to clean data, effectively turning deep learning into a robust form of kernel ridge regression.

The Problem: Memorizing Noise

Modern deep neural networks are "over-parameterized," meaning they have far more parameters than training examples. While this allows them to learn complex patterns, it also gives them the unwanted ability to fit completely random or "noisy" labels. If your dataset contains mislabeled images (e.g., a cat labeled as a dog), a standard network will eventually memorize that error, severely hurting its performance on real-world, clean data.

Practitioners often use "early stopping" to combat this, halting training before the model has time to memorize the noise. However, early stopping is a heuristic that is difficult to analyze theoretically and requires careful tuning. This paper seeks principled alternatives that are easy to implement and come with mathematical "generalization guarantees"—proofs that the model will perform well on clean data despite being trained on noisy labels.

Proposed Methods: RDI and AUX

The authors propose two intuitive methods to combat noise memorization:

  1. Regularization by Distance to Initialization (RDI): Instead of standard weight decay (which pulls weights toward zero), RDI adds a penalty term $\lambda \|\theta - \theta(0)\|^2$. This keeps the network parameters $\theta$ close to their random starting values $\theta(0)$.
  2. Auxiliary Variables (AUX): For every training example $i$, the authors add a trainable scalar $b_i$. The model tries to fit the noisy label $y_i$ using the sum of the neural network output and this variable: $f(\theta, x_i) + \lambda b_i$. At test time, the $b_i$ variables are discarded. The intuition is that the $b_i$ variables "absorb" the noise, leaving the neural network to learn only the consistent, clean patterns.

The "Aha!" Connection: The authors prove that in the "wide network" limit (the NTK regime), these two seemingly different methods are mathematically equivalent to Kernel Ridge Regression. This connects modern deep learning back to classical, well-understood statistical methods.

Key Insights and Results

The researchers evaluated their methods on three increasingly complex settings: binary MNIST, binary CIFAR-10, and full 10-class CIFAR-10 using a ResNet-34 architecture. In all cases, they manually corrupted the training labels with varying levels of noise (from 0% to 60%) to test robustness.

Test error across varying noise levels demonstrates that RDI and AUX perform comparably to optimal early stopping, and vastly outperform standard Gradient Descent.
  • Equivalence in Practice: In wide networks, RDI and AUX performed almost identically, confirming the authors' theoretical proof of their equivalence.
  • Noise Absorption: The AUX method is particularly clever. Because the model wants to minimize loss, it uses the $b_i$ terms to explain away labels that don't fit the general pattern. This prevents the weights of the actual neural network from twisting themselves into knots to fit outliers.
  • Theoretical Generalization: The authors provide a bound on the generalization error that depends on the clean labels, even though the model never sees them. This suggests that as long as the underlying "true" function is smooth, these regularizers will find it.

Stability and Outperforming Early Stopping

Unlike standard training, where test accuracy often peaks and then crashes as the model starts over-fitting noise, the validation accuracy with RDI/AUX is remarkably stable. This removes the guesswork involved in timing early stopping.

Validation accuracy over time on CIFAR-10. Standard MSE and CCE models overfit to noise (accuracy drops), while the AUX method maintains stable, high performance.
  • Outperforming Early Stopping: On CIFAR-10 with 60% label noise, a standard ResNet-34 with early stopping achieved ~79% accuracy. Using the AUX method, the same network reached ~83.9% accuracy (using MSE loss).
  • Robustness to Loss Functions: While the theory focuses on Mean Squared Error (MSE), the experiments showed that the AUX method works exceptionally well with standard Categorical Cross-Entropy (CCE) loss too.
  • Weight Movement: By tracking how much weights moved during training, the authors found that RDI and AUX significantly reduced the distance weights traveled, keeping the model in a more stable, generalizable regime.

Limitations and Future Work

  • The NTK Gap: The theoretical proofs rely on the Neural Tangent Kernel regime, which assumes the network is infinitely wide and weights move very little. However, the methods worked well on ResNet-34, which likely operates outside this regime. Explaining why it works so well for ResNets remains an intriguing direction for future work.
  • Hyperparameter Sensitivity: The methods introduce a new regularization strength parameter $\lambda$. While the paper shows the methods are robust, finding the optimal $\lambda$ still requires a grid search or validation set.
  • Computational Overhead: The AUX method requires storing one extra variable per training example. While negligible for MNIST, this could become a memory consideration for extremely large-scale datasets with billions of samples.

Summary updated

Abstract

Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data. Such over-fitting ability hinders generalization when mislabeled training examples are present. On the other hand, simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios, a phenomenon not theoretically understood. This paper proposes and analyzes two simple and intuitive regularization methods: (i) regularization by the distance between the network parameters to initialization, and (ii) adding a trainable auxiliary variable to the network output for each training example. Theoretically, we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels. Our generalization analysis relies on the connection between wide neural network and neural tangent kernel (NTK). The generalization bound is independent of the network size, and is comparable to the bound one can get when there is no label noise. Experimental results verify the effectiveness of these methods on noisily labeled datasets.
BibTeX
@inproceedings{hu2020simple,
  author = {Wei Hu and Zhiyuan Li and Dingli Yu},
  title = {{Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=c31KSjnZFC0},
}
Figure