Mitigating the Memorization of Noisy Labels in Deep Networks

This direction is split into other directions: label noise memorization (overfitting to noisy labels). Its last version stays readable and citable here.

Version 1 of 1455 papers479 conceptsfrozen 2026-10-04

Deep neural networks are notoriously "greedy" learners. Because of their high parameter capacity, they possess the mathematical flexibility to fit almost any dataset, including those where the labels are entirely random. While this capacity is what allows them to master complex tasks like facial recognition or medical imaging, it becomes a significant liability when training data is "noisy"—containing samples that are incorrectly or ambiguously labeled.

The core challenge in this research area is the memorization effect: the tendency of deep learning models to eventually treat errors as ground-truth facts, distorting their internal representations to accommodate noise.

The Clean First Noise Later Phenomenon

A recurring observation across the literature is that deep neural networks do not learn all data points at the same rate. During the early stages of training, models tend to prioritize simple, consistent patterns that represent the true underlying distribution of the data. As Xiaobo Wang, Hailin Shi, and colleagues note in Co-Mining: Deep Face Recognition With Noisy Labels, models learn from "clean" data first before eventually reaching a tipping point where they begin to memorize complex noise and outliers.

This temporal separation provides a window of opportunity. If a model's training can be interrupted or its learning objective modified before this "memorization phase" takes hold, the model can retain its ability to generalize to new data. Without such mechanisms, the model eventually force-fits the noise, leading to what Timo Kaiser and Christoph Reinders describe in Blind Knowledge Distillation for Robust Image Classification as a degradation of the model's internal structure.

Feature Distortion and Misleading Justifications

Why does memorization hurt performance so much? It isn't just that the model gets a few specific examples wrong; it's that the model warps its logic to justify those errors.

In Estimating Noisy Class Posterior with Part-level Labels for Noisy Label Learning, Rui Zhao and Bin Shi explain that models often "over-focus" on misleading features to justify an incorrect annotation. For instance, if a picture of a cat is incorrectly labeled "dog," the model might fixate on a specific patch of background grass and decide that this specific texture is a defining characteristic of a dog.

This is further evidenced in the work on Self-supervised Adversarial Noisy Masking (SANM), which shows that training on mislabeled data causes a model's activation maps to shift away from the primary object and toward irrelevant background features. This "distraction" prevents the network from learning the robust, semantic features necessary for real-world generalization.

Strategies for Robustness

To combat this, researchers have developed several distinct families of techniques:

1. Multi-Step Data Cleansing

One intuitive approach is to treat the dataset as a "dirty" resource that needs scrubbing. Kong Aik Lee and Ruijie Tao demonstrated in Speaker recognition with two-step multi-modal deep cleansing that by using auxiliary information (like visual data to verify audio speakers), researchers can filter out the "garbage" data before it has a chance to corrupt the model.

2. Peer Review and Co-Training

Rather than relying on a single model, frameworks like Co-Mining use two neural networks that "peer-review" each other. By exchanging high-confidence samples and identifying discrepancies between their predictions, the system can isolate noisy data points and prevent the error loops that lead to memorization.

3. Joint Optimization and Label Correction

Instead of treating labels as fixed truths, some researchers treat them as learnable variables. In Joint Optimization Framework for Learning With Noisy Labels, Daiki Tanaka and Toshihiko Yamasaki propose alternating between updating the model's weights and correcting the dataset's labels. This allows the model to "fix" the mistakes in its own training set as it learns the true underlying patterns.

4. Robust Loss Functions

Standard loss functions like Cross-Entropy are highly sensitive to outliers because they heavily penalize any deviation from the provided label. Qi Liu introduced the Skeptical Loss in Skeptical Deep Learning with Distribution Correction, which uses an adaptive transition matrix to make the model "skeptical" of labels that contradict its current understanding, effectively slowing down the memorization of suspicious samples.

The Path Forward: Data-Centric Diagnostics

Recent shifts in the field suggest a move toward "training-free" or "zero-shot" diagnostics. As Zhaowei Zhu and Yang Liu argue in Detecting Corrupted Labels Without Training a Model to Predict, we can often identify label errors simply by looking at the neighborhood information of data features. By moving from a learning-centric to a data-centric perspective, researchers are finding ways to break the circular dependency where a model becomes unable to distinguish between legitimate patterns and the noise it has already begun to memorize.

Papers in this version

Showing 60 of 455 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mitigating_the_memorization_of_noisy_lab_7677c0dd,
  title = {Mitigating the Memorization of Noisy Labels in Deep Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-the-memorization-of-noisy-labels-in-deep-networks/rdr_7677c0dda3dc43ee900ee96c884d6f7a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_mitigating_the_memorization_of_noisy_lab_7677c0dd_v1,
  title = {Mitigating the Memorization of Noisy Labels in Deep Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-the-memorization-of-noisy-labels-in-deep-networks/rdr_7677c0dda3dc43ee900ee96c884d6f7a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}