Small-Loss Sample Selection for Robust Deep Learning

26 concepts19 papers1 proposal

Deep neural networks possess an extraordinary capacity to memorize data. While this expressiveness allows them to learn complex patterns, it also makes them highly vulnerable to training errors: if a dataset contains corrupted or incorrect labels, a standard network will eventually memorize those mistakes, severely degrading its ability to generalize.

However, neural networks do not learn everything at once. They exhibit a distinct memorization effect (or early learning phenomenon), prioritizing simple, clean, and consistent patterns before brute-forcing the memorization of noisy outliers. This temporal separation forms the foundation of small-loss sample selection, a powerful paradigm for training robust models in imperfect data environments.

The Core Heuristic: Loss as a Cleanliness Proxy

During the initial stages of training, a model optimized via standard loss functions (such as cross-entropy) will naturally fit the clean data first. Consequently, correctly labeled samples quickly yield low loss values, while mislabeled samples retain high loss values. As surveyed by Filipe R. Cordeiro et al. in A Survey on Deep Learning with Noisy Labels: How to train your model when you cannot trust on the annotations?, we can exploit this behavior by treating training loss as a proxy for label reliability.

[ Noisy Dataset ] ---> ( Compute Loss Early in Training )
                             |
                             v
               [ Sort Samples by Loss Value ]
               /                            \
              /                              \
     ( Low Loss Subsets )           ( High Loss Subsets )
              |                              |
              v                              v
     [ Keep: Clean Data ]          [ Filter Out: Noisy Data ]

By sorting the samples within a mini-batch by their current loss and only performing gradient updates on a specified percentage of the lowest-loss instances, a system can effectively lock noisy data out of the optimization process.

This basic principle has been successfully adapted across diverse domains. For instance, in self-supervised speaker recognition, Kong Aik Lee et al. developed Loss-gated Learning (LGL) in Self-supervised Speaker Recognition with Loss-gated Learning, applying a dynamic loss threshold to exclude unreliable pseudo-labels generated by unsupervised clustering. Similarly, Shikun Li et al. demonstrated in Coupled-View Deep Classifier Learning from Multiple Noisy Annotators that prioritizing low-loss instances allows classifiers to withstand conflicting annotations from crowdsourced data without performance degradation.

The Confirmation Bias Trap and Peer Learning

While the small-loss heuristic is intuitive, a naive single-model implementation suffers from a fatal flaw: confirmation bias. If a network selects its own training samples based on its own internal state, it will naturally favor samples that confirm its existing misconceptions. Any early mistake or systematic bias the network develops will be reinforced, causing it to drift and eventually memorize a biased subset of the noise.

To break this self-reinforcing loop, researchers introduced peer-learning frameworks. In their seminal work Co-teaching: Robust training of deep neural networks with extremely noisy labels, Bo Han 0003 et al. proposed Co-teaching. Instead of one network acting as judge and jury for itself, Co-teaching maintains two distinct neural networks simultaneously.

[ Mini-Batch ] ---> Network A ---> ( Selects Small-Loss Samples ) ---> Trains Network B
               ---> Network B ---> ( Selects Small-Loss Samples ) ---> Trains Network A

In each iteration, Network A identifies its small-loss samples and feeds them to Network B to learn from, and vice versa. Because the two networks have different initializations and process data in different sequences, their error patterns remain somewhat independent, allowing them to filter out each other's noise through a "cross-pollination" of knowledge. This strategy has proven highly versatile; for example, Jiazhan Feng et al. adapted it to conversational AI in Learning a Matching Model with Co-teaching for Multi-turn Response Selection in Retrieval-based Dialogue Systems to filter out false negatives in automatically labeled dialogue datasets.

However, as training progresses, peer networks can still converge toward a biased consensus. To prevent this collapse, Xingrui Yu et al. introduced Co-teaching+ in How does Disagreement Help Generalization against Label Corruption?. This framework forces the peer networks to only learn from small-loss samples where they disagree on the predicted label, maintaining functional diversity between the models. Alternatively, Hongxin Wei et al. proposed JoCoR (Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization), which simplifies the architecture by utilizing a joint loss function. JoCoR selects small-loss examples based on the combined agreement of multiple networks, explicitly encouraging them to converge on clean patterns while filtering out high-loss noise.

Broad Applications Across Machine Learning

The utility of small-loss selection extends far beyond standard image classification, serving as a foundational tool for managing uncertainty across various complex tasks:

Even outside of label noise mitigation, the core idea of filtering by difficulty can accelerate architecture design. In Less is More: Proxy Datasets in NAS approaches, Brian Bernhard Moser et al. use low reconstruction or classification loss from a secondary model to filter out complex data, creating clean "proxy datasets" that slash Neural Architecture Search times by 75%. Finally, in the context of machine unlearning, Deepak Ravikumar et al. (SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness) utilize a trusted subset identified via low cross-entropy loss to define a clean feature space, allowing for automated corrective updates without manual auditing.

Key Foundational Work

How does Disagreement Help Generalization against Label Corruption? (2019) — contributed 1 concept elements to this direction.
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018) — contributed 1 concept elements to this direction.
Learning a Matching Model with Co-teaching for Multi-turn Response Selection in Retrieval-based Dialogue Systems (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_small_loss_sample_selection_91e898a2,
  title = {small-loss sample selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/small-loss-sample-selection/rdr_91e898a25dad4949ab2097c9dd991407},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_small_loss_sample_selection_91e898a2_v1,
  title = {small-loss sample selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/small-loss-sample-selection/rdr_91e898a25dad4949ab2097c9dd991407/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake