The Dynamics of Label Noise Memorization

63 concepts59 papers1 proposal

Deep neural networks owe their remarkable success to overparameterization. By utilizing millions or billions of parameters, these models can approximate highly complex, non-linear functions. However, this massive representational capacity is a double-edged sword: if a network is flexible enough to learn any underlying pattern, it is also flexible enough to perfectly memorize random noise and incorrect training labels.

When datasets contain corrupted or incorrect annotations—a common occurrence in real-world, web-scrawled, or crowdsourced data—high-capacity models face a fundamental trade-off between low approximation error and high estimation error, as noted by Cheng et al.. Instead of ignoring uninformative data or treating errors as outliers, standard deep networks deploy their expressive power to warp their decision boundaries around mislabeled points, severely degrading their ability to generalize to clean, unseen test data.

The Temporal Split: Early Learning vs. Late Memorization

To understand why label noise memorization occurs—and how to stop it—we must look at the temporal dynamics of gradient descent. Neural networks do not learn all patterns at the same rate. Instead, they follow a distinct trajectory: they learn clean, structurally coherent patterns early in training before transitioning to memorizing idiosyncratic noise patterns later on (Han et al.; Liu et al.).

Training Progress:
[ Epoch 0 ] ----------------------------> [ Early Epochs ] ----------------------------> [ Late Epochs ]
  Initial state                            Learning simple, shared features               Forcing weights to fit noise
  High loss                                (Clean data fits naturally)                    (Decision boundaries warp)

Why does this happen? Clean data points share common underlying features (e.g., visual traits of a "dog"). When the network processes these points, their gradients align and reinforce each other, leading to rapid optimization along those feature dimensions. Conversely, noisy labels are typically isolated or contradictory. Their gradients cancel each other out early on.

It is only in the later stages of training, once the major structural patterns have been learned and the remaining loss comes from these stubborn outliers, that the optimization process forces the network's weights to adjust to these high-frequency exceptions. As a result, the model's internal complexity shifts. Abbas et al. demonstrate that a model's local effective dimension—a measure of its utilized capacity—increases significantly when it is forced to memorize randomized or noisy labels, directly correlating with a failure to generalize.

This memorization masks the inherent difficulty of corrupted samples. As training persists, the loss on mislabeled points drops significantly, making standard loss-based error detection ineffective if evaluated on the training set (Mercier et al.; Klie et al.).

Designing Objectives that Resist Noise

Because standard Cross-Entropy (CE) loss aggressively minimizes error on every single sample, it actively encourages networks to exploit their capacity to fit noisy labels. To counteract this, researchers design robust loss functions that reduce the gradient push from high-error outliers.

A prominent example is the $L_q$ loss introduced by Sabuncu et al., which acts as a mathematical bridge between Cross-Entropy and Mean Absolute Error (MAE). While CE over-indexes on hard, potentially mislabeled examples, MAE treats all errors equally but suffers from slow convergence. The $L_q$ loss balances these two behaviors, maintaining training efficiency while bounding the influence of corrupted labels.

Alternatively, the authors of complementary loss frameworks combine multiple objectives to isolate noise (Wang et al.). They apply standard CE to high-confidence samples to drive fast feature learning, while applying a noise-robust loss to the remaining data to safely extract edge-case information without triggering memorization.

Restricting the Function Space

If excess capacity is what allows a network to memorize noise, another intuitive defense is to restrict the model's available function space or force it to learn simpler representations.

  • Sparsity and Representation Anchors: Goel et al. show that Monte Carlo Dropout acts as a powerful defense by forcing the network to learn sparser, less volatile representations, effectively stripping away the degrees of freedom required to over-explain noise. Similarly, Cheng et al. introduce representation regularizers that use self-supervised learning (SSL) as a structural anchor, forcing the network to respect the natural geometric clustering of the data rather than blindly fitting corrupted labels.
  • Parameter Shielding: Rather than regularizing the entire network, frameworks like those proposed by Meng et al. identify "safe parameters"—subnetworks or lottery tickets that capture generalizable features—and shield them from being corrupted by noisy updates during training on weakly labeled data.

Exploiting Dynamics for Sample Selection and Correction

Because clean samples are mastered early and noisy samples are memorized late, the training trajectory itself provides a clear signal for identifying which labels are incorrect.

Instead of treating labels as static targets, frameworks like PENCIL treat labels as learnable probability distributions (Yi et al.). By observing how a network's predictions fluctuate or stabilize over time, algorithms can dynamically update the labels themselves. For instance, Song et al. leverage prediction consistency across epochs to selectively "refurbish" unclean samples, turning corrupted data into valid training signals.

Similarly, Zhao et al. analyze the "probability difference"—the gap between the predicted probability of the assigned label and the next most likely class. Because a network struggles to confidently fit a noisy label during the early-to-mid stages of training, a narrow or volatile probability difference serves as a reliable diagnostic tool to down-weight suspicious samples before the network's capacity is fully deployed to memorize them.

Key Foundational Work

Joint Optimization Framework for Learning With Noisy Labels (2018) — contributed 1 concept elements to this direction.
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_label_noise_memorization_noise_memorizat_3e9aa73c,
  title = {label noise memorization (noise memorization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-noise-memorization-noise-memorization/rdr_3e9aa73c8b364c58ad0bd630818f61a7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_label_noise_memorization_noise_memorizat_3e9aa73c_v1,
  title = {label noise memorization (noise memorization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-noise-memorization-noise-memorization/rdr_3e9aa73c8b364c58ad0bd630818f61a7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake