Label Noise as a Regularizer for Robust Generalization

This direction is split into other directions: label noise injection. Its last version stays readable and citable here.

Version 1 of 187 papers112 conceptsfrozen 2026-10-04

In the classical view of machine learning, label noise is a nuisance—a source of error that degrades performance and must be cleaned or mitigated. However, a growing body of research suggests that in the era of overparameterized neural networks, intentionally injecting noise into labels can be a powerful tool for regularization. By forcing a model to navigate a "dirty" training landscape, researchers have found they can prevent the memorization of specific samples, improve adversarial robustness, and help models generalize to unseen data.

The Interpolation Trap

Modern neural networks are often large enough to achieve zero training error, a phenomenon known as interpolation. While this is often harmless for standard accuracy, Konstantin Donhauser and colleagues in Interpolation can hurt robust generalization even when there is no noise demonstrate that interpolation is actively harmful to robust generalization. When a model perfectly fits every training point, it often creates complex, high-frequency decision boundaries to accommodate every outlier.

Injecting label noise breaks this path to simple interpolation. By randomly flipping labels, the training process becomes a "moving target." This prevents the model from settling into the dense, overfitted solutions typical of large-batch gradient descent, as noted by Jeff Z. HaoChen in Shape Matters: Understanding the Implicit Bias of the Noise Covariance. Instead, the noise introduces a parameter-dependent stochasticity that biases the network toward simpler, low-norm solutions that capture the underlying signal rather than the fluctuations.

Occupying Memorization Capacity

Why does adding more noise help a model deal with existing noise? The intuition lies in the finite "memorization capacity" of a neural network.

In Open-set Label Noise Can Improve Robustness Against Inherent Label Noise, Hongxin Wei and Bo An introduce Out-of-Distribution Noise (ODNL). By adding auxiliary images and assigning them labels that change every epoch, they create a "perpetual learning task." This occupies the model's capacity, preventing it from successfully memorizing the actual errors (inherent noise) in the primary dataset.

Similarly, Lei Feng shows in Partial-label Learning with Mixed Closed-set and Open-set Out-of-candidate Examples that assigning random labels to unknown classes diverts the network's capacity toward "harmless noise." This prevents the model from finding meaningful but incorrect patterns in open-set data that would otherwise degrade the learning of known classes.

Stochastic Perturbation as a Universal Regularizer

The mechanism of intentional label disturbance has been applied across various domains:

Noise as a Computational Resource

Biologically inspired models often treat noise not as an error, but as a resource. Elahe Arani and Bahram Zonooz explore this in Improving Generalization and Robustness with Noisy Collaboration in Knowledge Distillation. By pairing perturbed labels with stable "soft targets" from a teacher model, they guide student models toward representations that ignore label errors while maintaining high performance.

This "benign noise" can even be tailored at the gradient level. Tingting Wu and Hao Zhang propose STGN in STGN: an Implicit Regularization Method for Learning with Noisy Labels in Natural Language Processing, which identifies likely incorrect labels and applies higher levels of gradient disturbance to them, preventing the model from locking onto those errors.

Why It Works: The Signal-to-Noise Perspective

The theoretical justification for these methods often centers on the signal-to-noise ratio (SNR). Wei Huang and colleagues prove in How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime? that injecting label noise suppresses the memorization of data-intrinsic noise.

By forcing the optimization process to handle constant label flips, the model is pressured to prioritize features that are consistently correlated with labels over many iterations. This "exponential growth" of the true signal relative to the noise allows the model to find the underlying structure of the data even when the initial data quality is poor. As Shokichi Takakura and Taiji Suzuki argue in their Mean-field Analysis, this process effectively learns a "data-dependent kernel" that is biased toward simpler, more generalizable functions.

Papers in this version

Showing 60 of 87 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_label_noise_as_a_regularizer_for_robust__534dc7a6,
  title = {Label Noise as a Regularizer for Robust Generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-noise-as-a-regularizer-for-robust-generalization/rdr_534dc7a6607a4890a07d2dcd540c857d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_label_noise_as_a_regularizer_for_robust__534dc7a6_v1,
  title = {Label Noise as a Regularizer for Robust Generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-noise-as-a-regularizer-for-robust-generalization/rdr_534dc7a6607a4890a07d2dcd540c857d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}