Deep neural networks possess a remarkable, yet problematic, ability to memorize data. While this allows them to learn complex patterns, it also means they will eventually memorize incorrect labels—often referred to as "noisy labels"—if trained long enough. This memorization ruins generalization, as the model treats random errors as ground-truth signals.
To combat this, researchers have developed a suite of strategies that move beyond standard supervised learning. These methods rely on a core intuition: neural networks learn simple, clean patterns before they learn complex, noisy ones.
The Small-Loss Trick
The foundation of most noise-robust frameworks is the "small-loss" hypothesis. In the early stages of training, samples with correct labels typically result in lower loss values than samples with incorrect labels. By treating loss as a proxy for label quality, models can distinguish between clean and corrupted data.
Han et al. pioneered this with Co-teaching, which addresses a major flaw in single-model filtering: confirmation bias. If a single model filters its own data, it will keep selecting samples that confirm its existing errors. Co-teaching uses two networks that "teach" each other; each network identifies small-loss samples in a mini-batch and passes them to its peer for training. This cross-pollination prevents the error of one network from directly reinforcing itself.
Building on this, Wei et al. introduced JoCoR, which encourages two networks to agree on predictions. By combining a joint loss function with the small-loss selection, the authors show that models can maintain stability even when 80% of the labels are flipped, as the agreement signal acts as a powerful regularizer against noise memorization.
Reframing Noise as Missing Information
Rather than simply discarding noisy samples, some of the most effective methods treat the problem as a semi-supervised learning (SSL) task. If we can identify which labels are likely wrong, we can discard the labels but keep the images, treating them as unlabeled data.
Li et al. demonstrated this with DivideMix. The framework uses a Gaussian Mixture Model (GMM) to fit the loss distribution of the training data, dynamically splitting it into a labeled "clean" set and an unlabeled "noisy" set. By applying SSL techniques like MixMatch to the combined pool, the authors achieved state-of-the-art results even in scenarios where 90% of the labels were incorrect.
Kim further refined this splitting process with SplitNet, arguing that instead of using handcrafted heuristics like GMMs, a learnable module can better adapt to specific noise distributions, allowing for more precise separation in extreme noise regimes.
From Filtering to Active Correction
Discarding data is often inefficient. A more proactive approach is to "fix" the labels during training. This is particularly important for "feature-dependent" noise, where harder-to-classify images are more likely to be mislabeled.
Zhang et al. proposed Progressive Label Correction (PLC), which acts like a curriculum. The model starts by correcting labels it is most certain about and gradually moves toward the more ambiguous decision boundaries. This iterative refinement prevents the model from being overwhelmed by noise early on.
For even more complex scenarios, Zheng et al. utilized meta-learning to automate this process. Their Meta Label Correction framework uses a small, clean validation set to learn a "meta-corrector" that predicts the true label for noisy training samples. The authors found that this approach is significantly more robust than simple re-weighting, as it recovers the underlying signal rather than just dampening the noise.
Modeling the Transition
Another perspective involves modeling the noise itself as a probabilistic process. If we can estimate the probability that a clean label $y$ is flipped to a noisy label $\tilde{y}$, we can "invert" the noise.
Yang et al. simplified this by focusing on the transition from Bayes optimal labels to noisy labels. Because Bayes optimal labels are deterministic and can be inferred from the noisy distribution, this method reduces the complexity of the hypothesis space, making it much easier for deep networks to learn the noise transition matrix in real-world, instance-dependent noise scenarios.
Structural and Optimization Robustness
Beyond data-centric methods, structural changes to the learning process itself can provide resilience.
- Curriculum Learning: Meng et al. showed that Meta Self-Paced Learning can dynamically adjust the "age" or pacing of a model, ensuring it only tackles harder (and potentially noisier) samples once it has mastered the clean, easy ones.
- Sharpness-Awareness: Wang et al. demonstrated that seeking "flat" minima in the loss landscape—a technique known as Sharpness-Aware Minimization—inherently improves robustness to label noise, as noisy labels tend to create sharp, non-generalizable spikes in the loss surface.
- Joint Support Estimation: Lampert et al. argued for modeling the support of the joint distribution of inputs and labels rather than a strict decision boundary. This allows the model to treat mislabeled examples as outliers that fall outside the learned support, maintaining performance even when the majority of data is corrupted.