Deep neural networks possess a paradoxical ability: they are powerful enough to memorize completely random labels, yet when trained on real-world datasets, they consistently learn generalizable patterns first. This phenomenon—known as the early learning or temporal memorization gap—reveals that deep networks do not treat all data equally. Instead, they follow a predictable, two-stage learning trajectory that naturally separates clean, structured signals from stochastic noise over time.
Understanding why this temporal separation occurs, and how to exploit it, has become a cornerstone for training robust models on imperfect, real-world data.
The Two-Stage Learning Trajectory
When a network is trained on a dataset containing label noise (where some percentage of the annotations are incorrect), its optimization path undergoes two distinct phases.
Phase 1: Signal Alignment (Early Training)
Gradients from clean, consistent samples reinforce each other.
[Clean Sample: Dog] ---> \
[Clean Sample: Dog] ---> ======> Strong, coherent gradient update
[Noisy Sample: Car] ---> /
Phase 2: Noise Memorization (Late Training)
Clean samples are solved (loss ≈ 0). Isolated noisy gradients dominate.
[Clean Sample: Dog] ---> (Solved)
[Noisy Sample: Car] ---> ======> Overfitting to idiosyncratic errors
In the first stage, the network acts as if it is virtually blind to the noise. As demonstrated by Devansh Arpit et al. in A Closer Look at Memorization in Deep Networks, deep networks prioritize simple, shared structures over the brute-force memorization of outliers. During these initial epochs, validation performance on clean data climbs toward its peak, even while the training set contains significant corruptions.
Mathematically, this happens because of how gradients accumulate. As explained by Andi Han et al. in On the Role of Label Noise in the Feature Learning Process, the gradients pointing toward consistent, shared features across correctly labeled samples reinforce one another, building rapid momentum. Conversely, the gradients required to fit individual, mislabeled instances are idiosyncratic and point in conflicting directions, effectively canceling each other out early on.
This sequential detection of signal before noise is a fundamental property of gradient descent in multi-layered architectures. In a theoretical analysis of deep linear networks, Andrew K. Lampinen in An analytic theory of generalization dynamics and transfer learning in deep linear networks showed that the strongest data signals are captured first, followed by progressively weaker signals, and finally, random noise. Similarly, Yuan Cao in Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise proved that during early training, even infinitely wide networks behave like simple linear classifiers, focusing on global trends before utilizing their massive capacity to bend decision boundaries around complex noise.
Eventually, the network enters the second stage. Once the clean patterns are learned, the loss on those samples drops to near zero, silencing their gradients. The remaining, unmined loss comes entirely from the noisy labels. With no competing signals, the network begins using its capacity to memorize these errors, causing generalization performance to collapse.
Exploiting the Small-Loss Criterion
Because clean samples are learned first, they naturally exhibit lower training loss than noisy samples during the early stages of optimization. This insight gives rise to the "small-loss criterion": if we want to find the clean data in a corrupted dataset, we should look at the samples the model finds easiest to learn during its initial warm-up phase.
Initial Training Loss Distribution (After Warm-up)
Count
^
| /---\
| / \ /---\
| / Clean \ / Noisy \
| / Data \ / Data \
+--/-----------\------------------/-----------\---> Loss Value
|<- Low Loss (Keep/Correct) ->| |<- High Loss (Filter) ->|
This temporal gap allows researchers to build unsupervised noise filters. For instance, Eric Arazo et al. in Unsupervised Label Noise Modeling and Loss Correction demonstrated that by fitting a Beta Mixture Model to the loss distribution early in training, one can dynamically calculate the probability that any given sample is clean or noisy, correcting the loss function before memorization takes root.
However, relying on a single network to filter its own data creates a dangerous confirmation bias: the model may select samples that match its own early errors, compounding its mistakes. To solve this, Bo Han 0003 et al. proposed a cross-pollination strategy in Co-teaching: Robust training of deep neural networks with extremely noisy labels. By training two distinct networks simultaneously, each network identifies its own low-loss samples and feeds them to its peer. Because the networks initialize differently, they make independent errors, allowing them to filter out noise for each other effectively.
Beyond Filtering: Dynamic Label Correction
While throwing away high-loss samples protects the model, it discards valuable data. A more sophisticated approach leverages the model's early representations to correct the labels on the fly.
Because the network learns general features first, its predictions during the warm-up phase are often more accurate than the corrupted ground-truth labels themselves. In Self-Adaptive Training: beyond Empirical Risk Minimization, Chao Zhang et al. showed that by using a model's own evolving predictions as pseudo-labels, the system can capture early distilled knowledge to repair corruptions in the dataset before the parameters overfit to the noise.
Aligning the training objective with this natural timeline is crucial. As argued by Xiu-Chuan Li et al. in Dynamics-Aware Loss for Learning with Label Noise, static loss functions fail because they treat the optimization objective identically at epoch 1 and epoch 100. By designing losses that prioritize fitting early on and transition to strict robustness and label correction later, training frameworks can gracefully ride the wave of the network's natural learning dynamics.
Key Foundational Work
A Closer Look at Memorization in Deep Networks (2017) — contributed 2 concept elements to this direction.
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018) — contributed 1 concept elements to this direction.