Global Sample Selection for Robust Learning with Noisy Labels

15 concepts8 papers1 proposal

Deep neural networks are notorious for memorizing noisy (incorrect) labels. A standard remedy is the "small-loss trick": during training, the model treats samples with the lowest loss as clean data and discards or down-weights the rest. Typically, this selection happens locally within each mini-batch—for instance, keeping the top 70% lowest-loss samples in every batch.

However, this local approach suffers from a fundamental flaw: it assumes that noise is uniformly distributed across the entire dataset. In practice, noise levels fluctuate wildly from one mini-batch to the next. To solve this, a growing body of research shifts the paradigm from local mini-batch filtering to global sample selection, leveraging bounded divergence metrics to create stable, dataset-wide thresholds.

The Pitfall of Mini-Batch Noise Variance

When we filter noisy data within isolated mini-batches, we assume an idealized scenario where every batch contains the exact same proportion of clean and corrupted labels. As shown by Tong et al. in IdentifyMix: An efficient two-stage learning approach to combating label noise, this assumption is frequently violated in real-world datasets.

Imagine a dataset with an average noise rate of 30%. If a specific mini-batch randomly contains 60% noisy labels, a fixed 70% local selection threshold forces the model to treat at least 30% corrupted samples as clean. Conversely, if a mini-batch is exceptionally clean and contains only 5% noise, the same threshold needlessly discards 25% of perfectly valid, high-quality training data.

This batch-level selection bias introduces massive gradient variance. As argued by Zhao et al. in P-DIFF: Learning Classifier with Noisy Labels based on Probability Difference Distributions, relying purely on local batch statistics causes severe jitter and unstable training dynamics because the selection criteria constantly shift depending on the local neighborhood of samples.

Shifting to Global Distribution Thresholding

To escape the local variance trap, the selection process must be conducted across the entire dataset distribution rather than within isolated batches. This shift ensures that a sample is judged by an absolute standard of cleanliness rather than how it compares to its immediate neighbors in a mini-batch.

In Web-Supervised Network with Softly Update-Drop Training for Fine-Grained Visual Classification, Zhang et al. demonstrate that global sample selection prevents gradient instability by maintaining a consistent filtering threshold across training epochs. Instead of adjusting the threshold to fit the local noise ratio of a small batch, the model evaluates data purification globally.

This macro-level view is further refined by Cai et al. in Group benefits instance for data purification, who calculate dynamic purification thresholds based on the dataset-wide mean and standard deviation of sample metrics. This global approach mitigates the negative impacts of noise-ratio imbalances that naturally occur when hardware constraints limit batch sizes.

A parallel intuition exists in broader noise-regime optimization: Gnanasambandam et al. demonstrate in One Size Fits All: Can We Train One Denoiser for All Noise Levels? that uniform allocation strategies fail to handle continuous variations in data difficulty, reinforcing the necessity of optimization frameworks that look across the entire data distribution rather than assuming local uniformity.

Why Bounded Divergence Enables Global Selection

If global thresholding is mathematically superior, why isn't it always used? The challenge lies in the metric. Standard cross-entropy loss is unbounded ($0$ to $\infty$). As a model trains, its average loss drops significantly, meaning an absolute loss threshold of 0.5 might mean "noisy" in epoch 2 but "clean" in epoch 50. Tracking an unbounded, moving distribution globally usually requires complex probabilistic modeling, such as fitting Gaussian Mixture Models to the loss distribution at every epoch.

To bypass this complexity, recent architectures employ symmetric and bounded divergence metrics—most notably, Jensen-Shannon Divergence (JSD)—between the model's predictions and the given labels.

Standard Cross-Entropy (Unbounded: 0 to ∞)
[0]-------------------------------------------------------------> [Shifted by Epoch]

Jensen-Shannon Divergence (Bounded: 0 to 1)
|[0]=========================[Fixed Threshold]=========================[1]|

Because JSD is strictly bounded (typically between 0 and 1), it provides an absolute scale for label quality. As Karim et al. explain in UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning, a bounded metric offers a stable selection criterion that behaves predictably regardless of local batch fluctuations or the absolute scale of the network's outputs.

This mathematical boundedness allows for two powerful behaviors:

  1. Fixed Global Thresholding: In Jo-SRC: A Contrastive Approach for Combating Noisy Labels, Yao et al. utilize JSD to establish a fixed threshold across the entire dataset. Because the metric is absolute, the threshold adapts seamlessly to varying noise levels across training batches without accidentally discarding clean data or swallowing noise.
  2. Safe Adaptive Thresholding: As the model's feature representations mature, the global threshold can be updated dynamically. Tang et al. show in Combating Noisy Labels through Fostering Self- and Neighbor-Consistency that bounded metrics allow these adaptive thresholds to evolve safely. Because the metric cannot blow up, the system avoids catastrophic failure modes where the threshold shifts too rapidly, preventing the network from prematurely memorizing noise or starving itself of training data.

Key Foundational Work

Jo-SRC: A Contrastive Approach for Combating Noisy Labels (2021) — contributed 3 concept elements to this direction.
Web-Supervised Network with Softly Update-Drop Training for Fine-Grained Visual Classification (2020) — contributed 3 concept elements to this direction.
One Size Fits All: Can We Train One Denoiser for All Noise Levels? (2020) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_global_sample_selection_840a9d17,
  title = {global sample selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-sample-selection/rdr_840a9d17cd464cf99aa5192e1596b37c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_global_sample_selection_840a9d17_v1,
  title = {global sample selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-sample-selection/rdr_840a9d17cd464cf99aa5192e1596b37c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake