In an ideal training scenario, labels are ground truth. But in large-scale web-crawled datasets or crowdsourced tasks, labels are often "noisy"—meaning a significant portion of them are incorrect. While deep neural networks (DNNs) are surprisingly resilient to moderate noise, they eventually succumb to the "garbage in, garbage out" principle. When noise levels reach extreme thresholds—where 80% or even 98% of the labels are wrong—standard supervised learning collapses.
Recent research has moved beyond simple robustness toward active recovery, developing mechanisms that can extract a clean signal even when the majority of the data is corrupted.
The Small Loss Phenomenon
The foundation of most noise-robust algorithms is the observation that DNNs learn simple, clean patterns before they begin to memorize complex, noisy ones. This "early learning" phase means that in the initial stages of training, samples with low loss are much more likely to be correctly labeled than those with high loss.
Amos Storkey and colleagues demonstrate that "noisy early stopping" can prevent a model from overfitting to corrupt data. However, simply stopping early isn't enough for extreme noise. To truly recover the signal, we must distinguish between the distribution of "clean" losses and "noisy" losses. Eric Arazo popularized the use of Beta Mixture Models (BMM) to fit the loss distribution of a training batch. By treating the loss as a mixture of two populations, the model can estimate the probability that any given sample is mislabeled and dynamically weight its contribution to the gradient.
From Instance Confidence to Global Consistency
A major challenge in extreme noise is that instance-level checks become unreliable. If 90% of your neighbors are mislabeled, a local consensus will only reinforce the error. Shilin Xu 0003 and Ahmet İşcen address this by using graph-based consistency. Instead of looking at a single label, they look at the structural influence of "seed" examples across a feature graph, allowing clean information to propagate through the manifold even when surrounded by noise.
In natural language tasks, Wenkai Zhang argues that we must model noise as a global distribution. Their Hypergeometric Learning (HGL) framework moves away from individual confidence scores, which tend to be corrupted by the prevalence of errors, and instead uses global statistical properties to denoise distantly supervised data.
The Geometry of the Latent Space
When labels are too noisy to trust, researchers turn to the geometry of the model’s internal representations. Taehyeon Kim 0003 introduced FINE, a method that detects noisy instances by analyzing the eigenvectors of the feature matrix. The intuition is that clean samples, despite being the minority in extreme noise scenarios, often cluster along dominant principal components in the latent space, while noisy samples are scattered or follow different structural patterns.
Similarly, Krishanu Sarker utilizes feature-space density. By identifying high-density regions in the feature space, models can "abstain" from low-density regions where noise typically resides. This "plug-and-play" approach allows for denoising without changing the underlying architecture.
Self-Correction and Meta-Purification
The most advanced frontier in this area is "label refurbishment"—not just ignoring bad labels, but fixing them. Yuanpeng Tu and Yuxi Li developed the Decoupled Meta Label Purifier (DMLP), which separates the learning of features from the correction of labels. This prevents the model from falling into a "confirmation bias" loop where it simply reinforces its own initial mistakes.
Perhaps most surprisingly, Hanxuan Wang has shown that we don't even need a clean validation set to perform this correction. Their STCT framework uses a "noisy meta-learning" approach, proving mathematically that a noisy validation set can still provide a gradient that points toward the true, clean distribution. This allows models to self-correct in environments where human-verified data is entirely unavailable.
The Limits of Recovery
Despite these advances, a fundamental "signal-to-noise" limit remains. Nazmul Karim notes that while frameworks like UNICON can handle up to 98% noise by combining uniform selection with contrastive learning, the stability of training becomes precarious. As Nandyala Hemachandra points out, when noise becomes "class-conditional"—meaning the errors are not random but systematically biased—the model may eventually find it impossible to distinguish the intended concept from the pervasive corruption.