Deep neural networks possess an extraordinary capacity to fit complex functions. However, this overparameterization is a double-edged sword: when trained on datasets containing corrupted or incorrect annotations, a network will eventually memorize these errors. This memorization forces the model to learn convoluted decision boundaries to justify the noise, severely degrading its ability to generalize to clean, unseen data.
To build models that can learn from imperfect, web-scraped, or crowdsourced data, researchers must design strategies that disrupt this memorization process. Understanding why these strategies work requires looking closely at the training dynamics of deep networks and how we can manipulate their objectives.
The Early Learning Phenomenon
The saving grace of training on noisy data is that deep networks do not memorize mistakes immediately. Instead, they exhibit a distinct temporal pattern known as early learning. During the initial phases of optimization, neural networks prioritize learning the simple, regular, and prominent patterns that underlie the majority of the data—which typically correspond to the clean, correctly labeled samples. Only in later epochs do the networks begin to bend their decision boundaries to interpolate the noisy outliers.
As noted by Dawei Zhu et al. in Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification, identifying the exact inflection point where a model transitions from general feature learning to noise memorization is a critical challenge. Similarly, Uthman Jinadu et al. observe in Noise Correction on Subjective Datasets that networks consistently capture clean patterns before internalizing human errors. This temporal separation provides a strategic window: if we can intervene or alter the learning objective before the early learning phase ends, we can prevent memorization from taking hold.
Correcting the Loss Objective
Standard cross-entropy loss penalizes a model based on the absolute certainty of the provided labels. If a label is incorrect, cross-entropy forces the network to output a high-confidence prediction for that error, driving the weights toward memorization. One way to counter this is to mathematically adjust the loss function to account for potential corruption.
In Making Neural Networks Robust to Label Noise: a Loss Correction Approach, Alessandro Rozza et al. introduce Forward and Backward loss correction techniques. The core intuition relies on estimating a noise transition matrix $T$, where each entry $T_{ij}$ represents the probability that a true label $i$ has been mistakenly flipped to label $j$ in the dataset.
By multiplying the network's raw probability predictions by this matrix (Forward correction), the loss is computed against the expected noisy distribution rather than treating the corrupted label as an absolute truth. This aligns the optimization objective with the reality of the data collection process, removing the incentive for the network to distort its internal features to fit anomalies.
Alternatively, regularization can be used to prevent the network from producing the overconfident predictions required to memorize noise. Michal Lukasik et al. demonstrate in Does label smoothing mitigate label noise? that label smoothing acts as an effective denoising mechanism. By softening target probabilities away from hard 1s and 0s, it restrains the model's weights from expanding excessively to fit corrupted samples, effectively acting as a structural barrier against noise memorization.
Sample Selection and Cross-Training
Because of the early learning phenomenon, clean samples typically yield lower loss values than noisy samples during the initial training epochs. This observation motivates the "small-loss trick": treating low-loss instances as clean data and discarding or reweighting high-loss instances.
However, relying on a single model to select its own clean data creates a dangerous feedback loop known as confirmation bias. If a model develops an early misconception, it will repeatedly select errors that match that misconception, reinforcing its own mistakes.
To break this cycle, multi-model frameworks are employed. For instance, Zhen Qin et al. in Making Deep Neural Networks Robust to Label Noise: Cross-Training With a Novel Loss Function propose a dual-model cross-training strategy. By allowing two independent networks to "peer-review" each other's samples, the framework filters out high-loss anomalies before they can corrupt either network's weights, ensuring that only mutually verified data guides the optimization.
Shifting the Paradigm via Negative Learning
Standard training relies on Positive Learning (PL): instructing a model that an image is a specific class (e.g., "This is a dog"). When the label is wrong, PL directly forces the model to associate the image features with the wrong class.
To mitigate this risk, Youngdong Kim et al. introduce an alternative paradigm in NLNL: Negative Learning for Noisy Labels. Instead of optimizing the model to match a potentially corrupted label, Negative Learning (NL) trains the model on complementary statements: "This image is not a car."
Because there are many incorrect classes and only one true class, the probability that a randomly chosen negative label is actually correct is extremely low. This drastically reduces the chances of the model memorizing incorrect associations. As explored by Mengmeng Sheng et al. in Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learning, combining indirect negative supervision with partial label learning allows models to safely narrow down the true label space without ever directly overfitting to a specific corrupted annotation.
Meta-Learning for Dynamic Purification
When a small, verified "clean" dataset is available, models can leverage meta-learning to automatically discover how to handle the remaining noisy data. Instead of relying on fixed heuristics or estimated transition matrices, the network learns to weight or correct labels dynamically based on how well those adjustments improve performance on the clean validation set.
In Learning to Learn From Noisy Labeled Data, Junnan Li et al. utilize a meta-learning loop to optimize a label-correction network, ensuring that the generated targets prevent the primary model from overfitting to noise. Similarly, Qi Xie develops the Learning to Purify Noisy Labels via Meta Soft Label Corrector framework, which maps corrupted hard labels to clean soft labels by treating the purification process as a meta-optimization problem. By anchoring the training process to a small set of reliable ground-truth data, these meta-learning approaches effectively bypass the risk of noise memorization across massive, web-scraped datasets.