Supervised learning traditionally rests on a fragile assumption: that the training labels provided to a model represent an absolute, pristine ground truth. In practice, this assumption is routinely violated. Whether due to the automated scraping of massive web datasets (Yandong Guo et al. in MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition), human annotator fatigue and confusion (Ishan Jindal et al. in An Effective Label Noise Model for DNN Text Classification), or inherent biological and environmental ambiguity (Ata Kaban et al. in Classification of mislabelled microarrays using robust sparse logistic regression), label noise is pervasive.
When standard optimization objectives like cross-entropy or negative log-likelihood are exposed to corrupted labels, they force models to become overconfident in incorrect assignments. As a result, the model internalizes erroneous patterns rather than capturing the true underlying physical or semantic distribution (Niranjan Thakurdesai in Learning an Uncertainty-Aware Object Detector for Autonomous Driving). To build reliable systems, researchers have developed techniques that exploit the structural behavior of deep networks to filter, correct, and learn robustly from imperfect data.
The Memorization Effect as a Filter
Why can deep neural networks handle noise at all? The core intuition relies on a phenomenon known as the memorization effect: deep networks naturally learn simple, clean, and consistent patterns before they begin overfitting to complex, random noise.
In the early stages of training, clean examples—which share coherent features across the dataset—yield low loss values because the network easily maps their shared structures. Conversely, mislabeled examples appear as statistical outliers, resulting in high loss values. Bo Yuan leverages this dynamic in Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLMs-Powered Assistance to establish dynamic thresholds that isolate noisy samples based on their loss trajectories over time.
Early Training Epochs:
[Clean Data] ---> Shared Patterns ---> Low Loss ---> Model Learns First
[Noisy Data] ---> Random Outliers ---> High Loss ---> Model Memorizes Later
Building on this "small-loss" heuristic, Hongxin Wei et al. developed JoCoR (Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization). Instead of relying on a single network's potentially biased view, the framework trains two distinct networks simultaneously. By selecting only the small-loss examples where both networks agree on the predictions, the system prevents either model from memorizing the noise, effectively filtering out corrupted labels on the fly.
Correcting the Loss via Noise Transition Matrices
When filtering data is too aggressive, or when every sample must be preserved, researchers turn to loss correction. If we can model how labels flip mathematically, we can adjust the objective function so that the model optimizes for the true, latent distribution despite receiving corrupted inputs.
This is achieved by estimating a noise transition matrix, where each entry $P(\tilde{Y}=j \mid Y=i)$ represents the probability that a true label $i$ has been corrupted into an observed label $j$. Ishan Jindal et al. implement this by appending a specialized, non-linear noise-modeling layer directly onto a deep text classifier (An Effective Label Noise Model for DNN Text Classification). This layer learns the statistics of annotator errors during training, allowing the network's core representations to remain uncorrupted.
However, applying a blanket correction across an entire dataset risks distorting perfectly clean samples. To resolve this trade-off, Yuri Grinberg et al. propose a selective approach called D&C-Net (Detect and Correct: A Selective Noise Correction Method for Learning with Noisy Labels). Their method analyzes the loss distribution to isolate only the highly suspicious samples, applying the noise transition matrix exclusively to those flagged instances while leaving the clean data intact.
Label Refinement and Calibration
Rather than just filtering samples or adjusting losses, an alternative strategy is label refinement: dynamically correcting corrupted labels into their true latent states during the training loop.
In multi-label classification, where errors are compounded across multiple tags, Jiankang Deng et al. introduce Holistic Label Correction (Holistic Label Correction for Noisy Multi-Label Classification). Their framework uses the model's own evolving predictions alongside a "sanity check" based on known label co-occurrences to fix corrupted tags. For instance, if an image is labeled with "indoor" and a noisy tag "car," the system uses the low co-occurrence probability to flag and correct the error.
When dealing with continuous label distributions rather than hard categories, Liang He et al. propose GCIA (Generative Calibration of Inaccurate Annotation for Label Distribution Learning). GCIA uses a generative model based on the principle that samples with similar features should share similar label distributions, recovering the true latent distributions from error-prone human annotations.
To unify these disparate strategies, Congyu Qiao et al. developed ULAREF (ULAREF: A Unified Label Refinement Framework for Learning with Inaccurate Supervision), which iteratively refines labels by combining global reliability detection with local model enhancements, providing a general framework for handling diverse forms of label corruption.
Robust Data Pruning and Geometric Medians
As datasets balloon in size, researchers use data pruning to select a small, highly informative subset of data to slash training costs. However, standard pruning techniques often fail under label noise because they mistake highly noisy examples for highly informative "hard" examples, leading to the selection of corrupted data.
To counter this, Dongmin Park et al. propose a pruning strategy that explicitly maximizes re-labeling accuracy (Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracy), ensuring that the pruned subset consists of truly informative samples rather than uncurated noise.
Another elegant solution relies on robust statistics: instead of matching the mean of a subset to the mean of a potentially corrupted dataset, algorithms can match the subset mean to the dataset's geometric median (Geometric Median (GM) Matching for Robust Data Pruning).
$$\arg\min_{\mu} \sum_{i=1}^{n} \|x_i - \mu\|_2$$
Because the geometric median minimizes the sum of Euclidean distances rather than squared distances, it possesses a high breakdown point of 0.5. This mathematical property ensures that the selected training subset remains representative of the clean underlying distribution, even if up to half of the original labels are completely corrupted.
Key Foundational Work
Boosting Applied toe Word Sense Disambiguation (2000) — contributed 1 concept elements to this direction.
An Empirical Study of the Noise Impact on Cost-Sensitive Learning (2007) — contributed 1 concept elements to this direction.