Mitigating Label Noise Memorization in Deep Neural Networks

This direction is split into other directions: label noise memorization (noise memorization). Its last version stays readable and citable here.

Version 1 of 1127 papers140 conceptsfrozen 2026-10-04

Deep neural networks are remarkably expressive. This high capacity allows them to learn complex patterns, but it also creates a fundamental vulnerability: they are "too good" at their jobs. If you provide a dataset where 40% of the labels are completely random, a standard high-capacity model will eventually achieve near-perfect training accuracy by simply memorizing the mistakes.

This phenomenon, often called the memorization paradox, creates a significant hurdle for training on real-world data, which is frequently "noisy" due to automated scraping or low-quality crowdsourcing. Research in this area focuses on a central observation: models do not learn everything at once. They learn simple, generalizable patterns first, and only later do they pivot to memorizing individual, noisy exceptions.

The Two Stages of Learning

The most critical intuition in this field is that learning happens in phases. As Xingjun Ma and colleagues demonstrate in Dimensionality-Driven Learning with Noisy Labels, deep networks initially compress data into low-dimensional manifolds—essentially finding the "signal." Only in the later stages of training do they expand into high-dimensional subspaces to fit the "noise."

This "early learning" effect is a recurring theme. Qi Qian and others provide a theoretical basis for this in A Theoretical Analysis of Learning with Noisily Labeled Data, showing that noisy labels initially act as a form of regularization before the model eventually overfits them. This suggests that the timing of training is just as important as the architecture itself.

The Small Loss Criterion

Because models learn clean data first, clean samples typically exhibit much lower loss values during the early epochs of training. This observation has birthed a family of "small-loss" heuristics.

One of the most influential approaches is Co-teaching, introduced by Bo Han 0003 and colleagues. Instead of a single model trying to filter its own noise (which leads to self-confirmation bias), two networks are trained simultaneously. Each network identifies the samples with the lowest loss in its current batch and feeds them to its peer for training. This "cross-pollination" helps filter out noisy labels before they can be memorized.

Similarly, Lu Jiang developed MentorNet, which uses a secondary "teacher" network to learn a data-driven curriculum. The teacher learns to weight samples so the primary "student" focuses on likely correct labels, effectively automating the intuition that we should prioritize "easy" (low-loss) samples.

Managing Model Capacity

If memorization is a byproduct of high capacity, one solution is to restrict that capacity. However, simply making a model smaller often hurts its ability to learn the actual signal.

Instead, researchers look for ways to manage "effective" capacity. Zhaowei Zhu and Yang Liu proposed Mitigating Memorization of Noisy Labels via Regularization between Representations, which aligns supervised outputs with features learned through self-supervised learning. This forces the model to stay grounded in the underlying data distribution rather than drifting into the high-dimensional "noise" space.

Another clever intervention targets the loss function itself. Standard cross-entropy loss is mathematically unbounded; if a model's prediction strongly disagrees with a label, the loss becomes massive, forcing the model to update its parameters aggressively to "fix" the error. Hongxin Wei and colleagues introduced LogitClip, which caps the magnitude of the model's outputs. By clipping these "logits," they prevent the loss from exploding on noisy samples, effectively taking the wind out of the sails of memorization.

Alternative Learning Signals

Some researchers argue that the problem lies in the "positive" nature of standard learning: telling a model "this image is a dog." If the label is wrong, the model is forced to learn a falsehood.

In Negative Learning for Noisy Labels, Youngdong Kim suggests a different path: telling the model "this image is not a bird." Because there are many more incorrect categories than correct ones, a "negative" label is much less likely to be accidentally true even if the data is noisy. This prevents the model from over-committing to a single incorrect class.

Alternatively, the model can be given an "out." Sunil Thulasidasan introduced the Deep Abstaining Classifier, which allows the network to "abstain" from making a prediction on confusing samples. By penalizing abstention while rewarding correct guesses, the model naturally learns to ignore samples that look like outliers or noise.

Summary of the Shift

The research direction has moved from viewing label noise as a static data problem to viewing it as a dynamic optimization problem. By understanding that models prioritize signal over noise in the early stages of training, researchers have developed a suite of tools—from Prestopping strategies to adversarial feature alignment like DAT—that allow us to harness the power of deep learning without falling victim to its capacity for memorization.

Papers in this version

Showing 60 of 127 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mitigating_label_noise_memorization_in_d_b8f93f84,
  title = {Mitigating Label Noise Memorization in Deep Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-label-noise-memorization-in-deep-neural-networks/rdr_b8f93f847bb542eb92062f49fcf575c9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_mitigating_label_noise_memorization_in_d_b8f93f84_v1,
  title = {Mitigating Label Noise Memorization in Deep Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-label-noise-memorization-in-deep-neural-networks/rdr_b8f93f847bb542eb92062f49fcf575c9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}