Deep neural networks possess a double-edged sword: they are universal function approximators capable of learning complex patterns, but they are also "memorizers" that can fit even completely random labels. In real-world datasets—scraped from the web, crowdsourced, or generated via distant supervision—labels are rarely pristine. When a model treats these errors as ground truth, it eventually memorizes the noise, causing its ability to generalize to plummet.
Research in this area focuses on a central intuition: clean samples are easier to learn than noisy ones. By identifying and prioritizing these "clean" instances during training, models can build a robust internal representation before the noise has a chance to corrupt them.
The Small-Loss Hypothesis
The cornerstone of many robust training methods is the "small-loss" trick. Early in training, a network typically learns simple, consistent patterns (the signal) before it begins to overfit to outliers (the noise). Consequently, samples with low loss values are statistically more likely to be correctly labeled.
Sujay Sanghavi and Yanyao Shen formalized this in their Iterative Trimmed Loss Minimization (ITLM) framework Iteratively Learning from the Best. ITLM works by alternating between two steps: selecting the subset of samples with the lowest current loss and retraining the model only on that subset. This "self-cleaning" cycle allows the model to progressively refine its understanding of the true data distribution. Similarly, Andreas Maurer and colleagues Robust Unsupervised Learning via L-statistic Minimization proposed using L-statistics, which weight smaller losses more heavily and ignore the largest ones, creating a natural resilience to outliers in clustering and dimensionality reduction.
Peer-Teaching and Consensus
A major risk of using a single model to select its own training data is "confirmation bias"—the model may select samples that confirm its own initial mistakes, leading to an error-reinforcement loop. To break this cycle, researchers introduced multi-network paradigms.
In Co-teaching, Bo Han and Quanming Yao Co-teaching: Robust training of deep neural networks with extremely noisy labels proposed training two networks simultaneously. Each network identifies a set of low-loss samples and feeds them to its peer for training. Because different networks learn different functions and make different mistakes, this "cross-pollination" filters out noise more effectively than self-selection.
This idea was further refined by Hongxin Wei and Lei Feng in JoCoR Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization. Instead of just sharing samples, JoCoR encourages two networks to agree on their predictions through a joint loss function, selecting only the "small-loss" examples where both networks are confident and in agreement.
The Noise Rate Bottleneck
While sample selection is powerful, it faces a persistent hurdle: the noise rate hyperparameter. Most methods, including Co-teaching and ITLM, require the user to pre-specify the proportion of corrupted data (e.g., "discard the top 20% of high-loss samples").
As noted across several papers, such as GRID by Gencer Sumbul and Begüm Demir, if this estimate is wrong, the model either retains too much noise or discards valuable clean data. In real-world scenarios, the noise rate is rarely known.
To address this, Aleksandr Karakulev and Dave Zachariah introduced RLVI Adaptive Parameter-Free Robust Learning using Latent Bernoulli Variables. This approach treats the "cleanliness" of each data point as a latent variable, allowing the model to automatically estimate the noise level through variational inference rather than relying on a manual threshold.
Expanding the Scope: From Images to LLMs
The principles of robust sample selection are now being applied to more complex domains:
- Large Language Models: Xize Liang and Chao Chen developed ROPO ROPO: Robust Preference Optimization for Large Language Models to handle noisy human preference data, which often contains ambiguous or incorrect rankings that can derail model alignment.
- Metric Learning: Chang Liu and Han Yu proposed PRISM Noise-Resistant Deep Metric Learning With Ranking-Based Instance Selection, which uses a memory bank of historical features to identify and filter mislabeled samples in embedding spaces.
- Model Restoration: Zhihao Sui and Zhongquan Sui introduced COLUR COLUR: Confidence-Oriented Learning, Unlearning and Relearning, a framework that "unlearns" high-confidence incorrect information before relearning from refined labels.
By moving away from the assumption of perfect data and instead treating data quality as a dynamic variable to be estimated, these methods allow machine learning to scale into the messy, uncurated environments of the real world.