Deep neural networks are remarkably expressive. This high capacity allows them to learn complex patterns, but it also creates a fundamental vulnerability: they are "too good" at their jobs. If you provide a dataset where 40% of the labels are completely random, a standard high-capacity model will eventually achieve near-perfect training accuracy by simply memorizing the mistakes.
This phenomenon, often called the memorization paradox, creates a significant hurdle for training on real-world data, which is frequently "noisy" due to automated scraping or low-quality crowdsourcing. Research in this area focuses on a central observation: models do not learn everything at once. They learn simple, generalizable patterns first, and only later do they pivot to memorizing individual, noisy exceptions.
The Two Stages of Learning
The most critical intuition in this field is that learning happens in phases. As Xingjun Ma and colleagues demonstrate in Dimensionality-Driven Learning with Noisy Labels, deep networks initially compress data into low-dimensional manifolds—essentially finding the "signal." Only in the later stages of training do they expand into high-dimensional subspaces to fit the "noise."
This "early learning" effect is a recurring theme. Qi Qian and others provide a theoretical basis for this in A Theoretical Analysis of Learning with Noisily Labeled Data, showing that noisy labels initially act as a form of regularization before the model eventually overfits them. This suggests that the timing of training is just as important as the architecture itself.
The Small Loss Criterion
Because models learn clean data first, clean samples typically exhibit much lower loss values during the early epochs of training. This observation has birthed a family of "small-loss" heuristics.
One of the most influential approaches is Co-teaching, introduced by Bo Han 0003 and colleagues. Instead of a single model trying to filter its own noise (which leads to self-confirmation bias), two networks are trained simultaneously. Each network identifies the samples with the lowest loss in its current batch and feeds them to its peer for training. This "cross-pollination" helps filter out noisy labels before they can be memorized.
Similarly, Lu Jiang developed MentorNet, which uses a secondary "teacher" network to learn a data-driven curriculum. The teacher learns to weight samples so the primary "student" focuses on likely correct labels, effectively automating the intuition that we should prioritize "easy" (low-loss) samples.
Managing Model Capacity
If memorization is a byproduct of high capacity, one solution is to restrict that capacity. However, simply making a model smaller often hurts its ability to learn the actual signal.
Instead, researchers look for ways to manage "effective" capacity. Zhaowei Zhu and Yang Liu proposed Mitigating Memorization of Noisy Labels via Regularization between Representations, which aligns supervised outputs with features learned through self-supervised learning. This forces the model to stay grounded in the underlying data distribution rather than drifting into the high-dimensional "noise" space.
Another clever intervention targets the loss function itself. Standard cross-entropy loss is mathematically unbounded; if a model's prediction strongly disagrees with a label, the loss becomes massive, forcing the model to update its parameters aggressively to "fix" the error. Hongxin Wei and colleagues introduced LogitClip, which caps the magnitude of the model's outputs. By clipping these "logits," they prevent the loss from exploding on noisy samples, effectively taking the wind out of the sails of memorization.
Alternative Learning Signals
Some researchers argue that the problem lies in the "positive" nature of standard learning: telling a model "this image is a dog." If the label is wrong, the model is forced to learn a falsehood.
In Negative Learning for Noisy Labels, Youngdong Kim suggests a different path: telling the model "this image is not a bird." Because there are many more incorrect categories than correct ones, a "negative" label is much less likely to be accidentally true even if the data is noisy. This prevents the model from over-committing to a single incorrect class.
Alternatively, the model can be given an "out." Sunil Thulasidasan introduced the Deep Abstaining Classifier, which allows the network to "abstain" from making a prediction on confusing samples. By penalizing abstention while rewarding correct guesses, the model naturally learns to ignore samples that look like outliers or noise.
Summary of the Shift
The research direction has moved from viewing label noise as a static data problem to viewing it as a dynamic optimization problem. By understanding that models prioritize signal over noise in the early stages of training, researchers have developed a suite of tools—from Prestopping strategies to adversarial feature alignment like DAT—that allow us to harness the power of deep learning without falling victim to its capacity for memorization.