Deep neural networks possess a remarkable and sometimes problematic capability: they can memorize virtually any training dataset, even one with completely random labels. In real-world scenarios—where datasets are scraped from the web, crowdsourced, or generated by automated heuristics—label noise is an inevitable tax.
If a model has the capacity to memorize errors, how can we prevent it from doing so? The field of learning with noisy labels focuses on exploiting the structural differences between how neural networks learn true underlying patterns versus how they memorize stochastic errors.
The Race Between Learning and Memorization
To understand why noise-robust algorithms work, we must first understand the internal dynamics of a standard neural network during training. Neural networks do not learn all patterns at the same rate. They exhibit a "race" phenomenon: clean, coherent semantic patterns are learned early in training, while noisy, chaotic labels are memorized much later.
As Drory et al. demonstrated in On the Resistance of Neural Nets to Label Noise, a deep network's internal feature space naturally behaves like a K-Nearest Neighbors (K-NN) classifier. When label noise is uniformly distributed across a dataset, the true semantic neighbors of a data point overwhelm the isolated, erroneous label. The network easily ignores these randomly spread errors during the initial phases of optimization. However, if noisy labels become locally concentrated in the feature space, the network begins to treat them as valid sub-classes, warping its representation boundaries to accommodate the errors.
This temporal separation between learning clean patterns and memorizing noise provides a leverage point. If we can identify which samples are driving the loss at different stages of training, we can intervene before memorization corrupts the model.
Correcting the Loss Landscape
One of the most direct ways to handle label noise is to mathematically alter the loss function so that the network optimizes for the true, unobserved clean distribution rather than the corrupted training distribution. This is typically achieved using a noise transition matrix, $T$, where each entry $T_{ij} = P(\tilde{y} = j \mid y = i)$ represents the probability that a true label $i$ has been flipped to a noisy label $j$.
In Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach, Patrini et al. introduced two fundamental strategies to operationalize this: Forward and Backward loss correction.
Forward Correction:
[Clean Prediction (p)] ---> [Multiply by Transition Matrix (T)] ---> [Noisy Prediction (p*)] ---> [Loss with Noisy Label]
In Forward Correction, instead of comparing the model's clean prediction directly to the noisy label, the model's prediction is first multiplied by the transition matrix $T$ to simulate the noise process. The cross-entropy loss is then computed between this "intentionally corrupted" prediction and the noisy label. This aligns the optimization objective: the network is penalized based on how well its underlying clean hypothesis, when subjected to the real-world noise degradation, matches the observed noisy data.
Recovering the Matrix Geometrically
Historically, estimating $T$ required "anchor points"—perfectly clean examples of each class that the model could use as a baseline. Because true anchor points are rare in messy datasets, Li et al. proposed an end-to-end framework called VolMinNet in Provably End-to-end Label-noise Learning without Anchor Points.
The intuition relies on geometry: the observed noisy class posteriors live within a simplex whose vertices correspond to the true, clean class posteriors. By minimizing the volume of this simplex while ensuring it still encapsulates the noisy data points, the framework provably recovers both the true labels and the transition matrix simultaneously, removing the need for pristine reference data.
Sample Selection and Meta-Reweighting
Instead of modifying the loss globally, another family of techniques filters or reweights individual samples based on their likelihood of being clean.
Because networks learn clean patterns first, clean samples exhibit low loss early in training, while noisy samples maintain high loss until the network begins to memorize them. This split can be modeled explicitly. For instance, Grinberg et al. developed D&C-Net in Detect and Correct: A Selective Noise Correction Method for Learning with Noisy Labels, which analyzes the loss distribution mixture during training to dynamically separate clean data from corrupted data, applying loss correction exclusively to the detected noisy subset.
A more automated approach bypasses manual loss thresholds entirely by using meta-learning. As shown by the authors of Learning to Reweight Examples for Robust Deep Learning, a network can learn to weight its own training examples by maintaining a tiny, clean validation set. Before each standard optimization step, the model performs a meta-gradient step: it calculates how a temporary change in the weights of the current training batch would impact the loss on the clean validation set. Examples that push the model toward better validation performance receive higher weights; examples containing contradictory, noisy gradients are down-weighted in real-time.
Self-Ensembling and Label Refinement
Rather than treating labels as static entities that must be accepted or discarded, we can treat them as soft, evolving targets. As a model trains, its internal representations become highly robust to random perturbations. We can use the model's own growing consensus to progressively correct the dataset.
In SELC: Self-Ensemble Label Correction Improves Learning with Noisy Labels, Lu et al. leverage an exponential moving average (EMA) of historical model predictions to refine the training labels.
$$\tilde{y}_{\text{refined}}^{(t)} = \alpha \tilde{y}_{\text{refined}}^{(t-1)} + (1 - \alpha) P_{\text{model}}(y \mid x)$$
Because the EMA acts as a low-pass filter over the training epochs, it smooths out the sudden gradient spikes caused by individual noisy labels. Once the training hits a data-driven "turning point"—the moment right before memorization typically accelerates—the framework begins substituting the noisy ground-truth labels with these self-ensembled predictions, effectively self-cleaning the dataset from within.
Normalizing Feature Space Dynamics
When a model is forced to fit a noisy label, what actually happens to its internal representations? The gradients from the incorrect label pull the feature vector of the noisy sample away from its true semantic cluster and drag it toward the cluster of the incorrect target class. This warps the latent space, degrading the model's ability to generalize to clean test data.
Standard Training with Noise:
[True Cluster A] ------(Gradient Pull)------> [Noisy Sample] -------> [Incorrect Cluster B]
(Warped Latent Space)
With Wasserstein Distributional Normalization (WDN):
[True Cluster A] <---(Optimal Transport Flow)--- [Noisy Sample] [Incorrect Cluster B]
(Preserved Latent Space)
To counteract this geometric warping, Kwon et al. proposed Wasserstein Distributional Normalization (WDN) in Wasserstein Distributional Normalization For Robust Distributional Certification of Noisy Labeled Data. Instead of discarding high-loss samples, WDN uses optimal transport theory to calculate a Wasserstein gradient flow in the feature space. This flow mathematically pushes the feature distributions of high-loss, uncertain samples back toward the distribution of verified clean samples. By anchoring the feature distributions to known clean archetypes, the model can still extract useful structural information from the inputs of noisy samples without allowing their erroneous labels to distort the underlying geometry of the latent space.