Deep neural networks possess a fascinating property: they are highly expressive enough to memorize completely random noise, yet they are inherently inclined to learn simple, clean patterns first. When presented with a dataset containing both clean and noisy (incorrect) labels, a network does not overfit immediately. Instead, it undergoes what is known as the deep learning memorization effect—it prioritizes generalizable, regular features early in training before slowly adapting its parameters to fit chaotic, high-loss outliers.
This temporal separation creates a powerful window of opportunity. If we can monitor the model's training dynamics, we can catch it right as it finishes learning clean patterns but before it begins digesting the noise. This is the core intuition behind Dynamic Loss Thresholding (DLT). Rather than relying on static data cleaning or fixed hyperparameters, DLT algorithms use the model's own evolving state to dynamically separate clean samples from corrupted ones.
Why Static Thresholds Fail
A naive approach to filtering noisy labels might involve setting a fixed loss threshold: for example, discarding any sample with a cross-entropy loss greater than a predefined constant. However, as Bo Yuan et al. argue in Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning, static filtering strategies fail because the optimal threshold varies across different samples, classes, and training stages.
Early in training, even clean samples can exhibit high loss because the model has not yet developed strong representations. A static, strict threshold would prematurely discard these informative but difficult examples. Conversely, late in training, the model's internal confidence naturally increases, causing it to lower its loss on noisy samples as it begins to memorize them. A static threshold cannot adapt to this shifting baseline, leading to either the loss of valuable data or the retention of harmful noise.
The Mechanics of Evolving Thresholds
To align filtering with the model's learning trajectory, DLT dynamically adjusts the boundary between clean and noisy data. This evolution typically follows one of two paradigms: scheduled tightening or automated self-tuning.
Scheduled Tightening
The most straightforward way to exploit the memorization effect is to gradually increase the stringency of the filter over time. In Co-teaching: Robust training of deep neural networks with extremely noisy labels, Bo Han 0003 et al. introduced a dynamic drop rate schedule. Early in training, the threshold is permissive, allowing the model to observe the full data distribution and learn general features. As training epochs progress and the model's capacity to overfit increases, the schedule discards an increasing proportion of high-loss samples.
This strategy is similarly leveraged by Chen Li et al. in From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation, where a dynamic loss threshold progressively prunes high-loss pseudo-labels. By tightening the criteria as the model matures, the system ensures that the supervision becomes cleaner exactly when the model becomes vulnerable to memorizing errors.
Automated Self-Tuning
While scheduled functions work well when the dataset's noise rate is known beforehand, real-world data rarely provides such guarantees. To bypass this limitation, researchers design thresholds that compute themselves automatically based on real-time training statistics.
In Learning with Instance-Dependent Label Noise: A Sample Sieve Approach, Hao Cheng et al. developed a confidence-regularized "sample sieve." Instead of using a fixed drop rate, their mechanism calculates an adaptive threshold directly from the model's current class predictions, maintaining high precision without prior knowledge of the error distribution.
A remarkably elegant variant of this idea is explored by Xu Ouyang et al. in Efficient NLP Model Finetuning via Multistage Data Filtering. They demonstrate that a moving average of recent training losses can serve as an automated, self-evolving threshold. Because the moving average naturally drops as the model converges, the filtering mechanism automatically evolves alongside the model's competence, eliminating the need for manual, task-specific hyperparameter tuning.
[ Early Training ] --> Model learns simple, clean patterns first.
Loss threshold is high/permissive.
[ Mid Training ] --> Moving average of loss drops. Threshold tightens.
High-loss samples (potential noise) are filtered.
[ Late Training ] --> Filtered samples are isolated or re-evaluated,
preventing the network from memorizing noise.
Advanced Paradigms: Peer Review and Dynamic Recovery
While adjusting a single threshold is powerful, it faces a fundamental challenge: confirmation bias. If a model mistakenly learns an incorrect pattern, it will assign low loss to similar incorrect samples, creating a feedback loop that reinforces its own errors. To solve this, the community has introduced two key architectural extensions.
Cross-Pollination (Peer Review)
To break the self-reinforcing error loop, Bo Han 0003 et al. proposed a dual-network architecture. Instead of a single network filtering its own data, two independent networks are trained simultaneously. Each network evaluates a mini-batch, applies its dynamic loss threshold to select the "clean" items, and then feeds those selected samples to its peer network for the actual gradient update. Because the networks initialize differently and see different data orderings, their error patterns are distinct, allowing them to effectively filter out each other's noise.
Dynamic Sample Recovery
A major risk of aggressive thresholding is the permanent loss of valuable data. Hard, complex, or highly augmented samples often exhibit high initial loss and can easily be misclassified as noise early in training.
To prevent this permanent data loss, modern frameworks treat the threshold not as a permanent deletion tool, but as a temporary sieve. In the framework proposed in Improving Distantly Supervised Relation Extraction with Self-Ensemble Noise Filtering, samples that conflict with the current model state are set aside but remain available for re-evaluation in subsequent epochs. As the model's internal representations mature and stabilize, it can dynamically recover these previously discarded samples once its predictive confidence passes the evolving threshold, ensuring that long-term generalization is not sacrificed for short-term noise avoidance.
Key Foundational Work
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018) — contributed 1 concept elements to this direction.
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018) — contributed 1 concept elements to this direction.
A novel ranking method for multiple classifier systems. (2015) — contributed 1 concept elements to this direction.