TL;DR: This paper introduces two simple regularization techniques—Regularization by Distance to Initialization (RDI) and Auxiliary Variables (AUX)—that prevent over-parameterized neural networks from memorizing noisy labels. By leveraging the Neural Tangent Kernel (NTK) framework, the authors provide a theoretical guarantee that these methods generalize well to clean data, effectively turning deep learning into a robust form of kernel ridge regression.
The Problem: Memorizing Noise
Modern deep neural networks are "over-parameterized," meaning they have far more parameters than training examples. While this allows them to learn complex patterns, it also gives them the unwanted ability to fit completely random or "noisy" labels. If your dataset contains mislabeled images (e.g., a cat labeled as a dog), a standard network will eventually memorize that error, severely hurting its performance on real-world, clean data.
Practitioners often use "early stopping" to combat this, halting training before the model has time to memorize the noise. However, early stopping is a heuristic that is difficult to analyze theoretically and requires careful tuning. This paper seeks principled alternatives that are easy to implement and come with mathematical "generalization guarantees"—proofs that the model will perform well on clean data despite being trained on noisy labels.
Proposed Methods: RDI and AUX
The authors propose two intuitive methods to combat noise memorization:
- Regularization by Distance to Initialization (RDI): Instead of standard weight decay (which pulls weights toward zero), RDI adds a penalty term $\lambda \|\theta - \theta(0)\|^2$. This keeps the network parameters $\theta$ close to their random starting values $\theta(0)$.
- Auxiliary Variables (AUX): For every training example $i$, the authors add a trainable scalar $b_i$. The model tries to fit the noisy label $y_i$ using the sum of the neural network output and this variable: $f(\theta, x_i) + \lambda b_i$. At test time, the $b_i$ variables are discarded. The intuition is that the $b_i$ variables "absorb" the noise, leaving the neural network to learn only the consistent, clean patterns.
The "Aha!" Connection: The authors prove that in the "wide network" limit (the NTK regime), these two seemingly different methods are mathematically equivalent to Kernel Ridge Regression. This connects modern deep learning back to classical, well-understood statistical methods.
Key Insights and Results
The researchers evaluated their methods on three increasingly complex settings: binary MNIST, binary CIFAR-10, and full 10-class CIFAR-10 using a ResNet-34 architecture. In all cases, they manually corrupted the training labels with varying levels of noise (from 0% to 60%) to test robustness.
- Equivalence in Practice: In wide networks, RDI and AUX performed almost identically, confirming the authors' theoretical proof of their equivalence.
- Noise Absorption: The AUX method is particularly clever. Because the model wants to minimize loss, it uses the $b_i$ terms to explain away labels that don't fit the general pattern. This prevents the weights of the actual neural network from twisting themselves into knots to fit outliers.
- Theoretical Generalization: The authors provide a bound on the generalization error that depends on the clean labels, even though the model never sees them. This suggests that as long as the underlying "true" function is smooth, these regularizers will find it.
Stability and Outperforming Early Stopping
Unlike standard training, where test accuracy often peaks and then crashes as the model starts over-fitting noise, the validation accuracy with RDI/AUX is remarkably stable. This removes the guesswork involved in timing early stopping.
- Outperforming Early Stopping: On CIFAR-10 with 60% label noise, a standard ResNet-34 with early stopping achieved ~79% accuracy. Using the AUX method, the same network reached ~83.9% accuracy (using MSE loss).
- Robustness to Loss Functions: While the theory focuses on Mean Squared Error (MSE), the experiments showed that the AUX method works exceptionally well with standard Categorical Cross-Entropy (CCE) loss too.
- Weight Movement: By tracking how much weights moved during training, the authors found that RDI and AUX significantly reduced the distance weights traveled, keeping the model in a more stable, generalizable regime.
Limitations and Future Work
- The NTK Gap: The theoretical proofs rely on the Neural Tangent Kernel regime, which assumes the network is infinitely wide and weights move very little. However, the methods worked well on ResNet-34, which likely operates outside this regime. Explaining why it works so well for ResNets remains an intriguing direction for future work.
- Hyperparameter Sensitivity: The methods introduce a new regularization strength parameter $\lambda$. While the paper shows the methods are robust, finding the optimal $\lambda$ still requires a grid search or validation set.
- Computational Overhead: The AUX method requires storing one extra variable per training example. While negligible for MNIST, this could become a memory consideration for extremely large-scale datasets with billions of samples.
Summary updated