Robust Learning with Noisy Labels via Sample Selection

This direction is merged into another direction: hyperparameter sensitivity (noise rate sensitivity). Its last version stays readable and citable here.

Version 1 of 148 papers52 conceptsfrozen 2026-10-04

Deep neural networks possess a double-edged sword: they are universal function approximators capable of learning complex patterns, but they are also "memorizers" that can fit even completely random labels. In real-world datasets—scraped from the web, crowdsourced, or generated via distant supervision—labels are rarely pristine. When a model treats these errors as ground truth, it eventually memorizes the noise, causing its ability to generalize to plummet.

Research in this area focuses on a central intuition: clean samples are easier to learn than noisy ones. By identifying and prioritizing these "clean" instances during training, models can build a robust internal representation before the noise has a chance to corrupt them.

The Small-Loss Hypothesis

The cornerstone of many robust training methods is the "small-loss" trick. Early in training, a network typically learns simple, consistent patterns (the signal) before it begins to overfit to outliers (the noise). Consequently, samples with low loss values are statistically more likely to be correctly labeled.

Sujay Sanghavi and Yanyao Shen formalized this in their Iterative Trimmed Loss Minimization (ITLM) framework Iteratively Learning from the Best. ITLM works by alternating between two steps: selecting the subset of samples with the lowest current loss and retraining the model only on that subset. This "self-cleaning" cycle allows the model to progressively refine its understanding of the true data distribution. Similarly, Andreas Maurer and colleagues Robust Unsupervised Learning via L-statistic Minimization proposed using L-statistics, which weight smaller losses more heavily and ignore the largest ones, creating a natural resilience to outliers in clustering and dimensionality reduction.

Peer-Teaching and Consensus

A major risk of using a single model to select its own training data is "confirmation bias"—the model may select samples that confirm its own initial mistakes, leading to an error-reinforcement loop. To break this cycle, researchers introduced multi-network paradigms.

In Co-teaching, Bo Han and Quanming Yao Co-teaching: Robust training of deep neural networks with extremely noisy labels proposed training two networks simultaneously. Each network identifies a set of low-loss samples and feeds them to its peer for training. Because different networks learn different functions and make different mistakes, this "cross-pollination" filters out noise more effectively than self-selection.

This idea was further refined by Hongxin Wei and Lei Feng in JoCoR Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization. Instead of just sharing samples, JoCoR encourages two networks to agree on their predictions through a joint loss function, selecting only the "small-loss" examples where both networks are confident and in agreement.

The Noise Rate Bottleneck

While sample selection is powerful, it faces a persistent hurdle: the noise rate hyperparameter. Most methods, including Co-teaching and ITLM, require the user to pre-specify the proportion of corrupted data (e.g., "discard the top 20% of high-loss samples").

As noted across several papers, such as GRID by Gencer Sumbul and Begüm Demir, if this estimate is wrong, the model either retains too much noise or discards valuable clean data. In real-world scenarios, the noise rate is rarely known.

To address this, Aleksandr Karakulev and Dave Zachariah introduced RLVI Adaptive Parameter-Free Robust Learning using Latent Bernoulli Variables. This approach treats the "cleanliness" of each data point as a latent variable, allowing the model to automatically estimate the noise level through variational inference rather than relying on a manual threshold.

Expanding the Scope: From Images to LLMs

The principles of robust sample selection are now being applied to more complex domains:

By moving away from the assumption of perfect data and instead treating data quality as a dynamic variable to be estimated, these methods allow machine learning to scale into the messy, uncurated environments of the real world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_robust_learning_with_noisy_labels_via_sa_16fc7ccf,
  title = {Robust Learning with Noisy Labels via Sample Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-learning-with-noisy-labels-via-sample-selection/rdr_16fc7ccf504e4401b8390257a6bcc910},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_robust_learning_with_noisy_labels_via_sa_16fc7ccf_v1,
  title = {Robust Learning with Noisy Labels via Sample Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-learning-with-noisy-labels-via-sample-selection/rdr_16fc7ccf504e4401b8390257a6bcc910/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}