Deep neural networks are notorious memorizers. If a training dataset contains corrupted annotations—whether from low-cost crowdsourcing, automated web scraping, or system pipeline errors—a standard supervised model will eventually fit these mistakes, severely damaging its capacity to generalize to real-world data.
The intuitive first response to this problem is filtering: if a label is suspected to be wrong, discard the sample entirely. However, as highlighted by Filipe R. Cordeiro et al. in their comprehensive overview, A Survey on Deep Learning with Noisy Labels, hard filtering throws the baby out with the bathwater. An image of a cat incorrectly labeled as a "toaster" still contains rich visual features—edges, textures, and geometric structures—that define what an object or animal looks like. Deleting the sample permanently deprives the model of this underlying structural information.
The core insight of modern noise-robust learning is to decouple the data representation from its faulty annotation. Instead of discarding a suspicious sample, frameworks strip its unreliable label and reclassify it as unlabeled data. This simple perspective shift transforms a noisy supervised learning problem into a semi-supervised learning (SSL) task, allowing models to exploit the full volume of the dataset without being misled by its errors.
Separating Clean Anchors from Noisy Data
To treat noisy data as unlabeled, a system must first reliably distinguish between clean and corrupted annotations. How can a network identify an error without access to a ground-truth oracle?
Most frameworks leverage the small-loss hypothesis: deep neural networks tend to learn simple, clean patterns before memorizing complex, noisy exceptions. Early in the training process, correctly labeled samples yield low loss because they align with dominant data distributions, whereas mislabeled samples yield high loss because the network struggles to force them into the wrong category boundaries.
[ Noisy Dataset ] ──> ( Early Training / Small-Loss Sieve )
│
├─── Low Loss ──> [ Clean Pool ] (Keep Labels)
│
└─── High Loss ──> [ Unlabeled Pool ] (Strip Labels)
Researchers exploit this phenomenon using various statistical and architectural sieves:
- Loss Mixture Modeling: The widely adopted DivideMix framework fits a two-component Gaussian Mixture Model (GMM) over per-sample losses dynamically during training to divide the dataset into a clean set and a noisy set.
- Two-Stage Isolation: Yifan Ding et al. propose a framework in A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels that isolates a high-confidence clean subset in the first stage, establishing a reliable anchor before treating the remaining uncertain data as unlabeled.
- Temporal Stability: Rather than relying purely on instantaneous loss values, Duc Tam Nguyen et al. introduce SELF: Learning to Filter Noisy Labels with Self-Ensembling. This method identifies noise by monitoring how much a model's predictions fluctuate across training epochs. If the network's internal consensus consistently disagrees with the provided label over time, the label is dropped.
Learning from the Unlabeled Pool
Once the dataset is split, the training objective bifurcates. For the clean subset, standard supervised losses apply. For the noisy subset, the model ignores the original labels entirely and applies semi-supervised objectives—primarily consistency regularization and data interpolation.
Consistency regularization forces the network to output identical predictions for different augmented views of the same unlabeled image. As Hao Cheng et al. demonstrate in Learning with Instance-Dependent Label Noise: A Sample Sieve Approach, this constraint forces the network to map the sample into a stable feature space based entirely on its actual content. Because the original corrupted label is ignored, the network avoids the misleading gradient updates that would normally warp its decision boundaries.
Furthermore, data augmentation techniques like MixUp are often applied across the clean and unlabeled pools. Diego Ortego et al. utilize this in Multi-Objective Interpolation Training for Robustness To Label Noise, interpolating both the inputs and targets of clean and stripped samples. This mathematical smoothing prevents the network from forming sharp, memorized peaks around noisy data points, encouraging a continuous and robust representation space.
Progressive Label Rectification
The ultimate power of this paradigm is that it is self-reinforcing. As the model trains on the clean anchors and extracts robust features from the unlabeled pool, its classification performance improves. A smarter model can then revisit the unlabeled pool and actively repair the annotations it initially stripped.
This process, known as label rectification or refurbishment, replaces the original corrupted labels with high-confidence model predictions (pseudo-labels).
[ Unlabeled Pool ] ──> ( Enhanced Feature Representation ) ──> [ Pseudo-Label Generation ]
▲ │
└─────────────────── Rectified Data ───────────┘
In the domain of text processing, Kaijia Yang et al. show in Exploiting Noisy Data in Distant Supervision Relation Classification that treating noisy sentences as unlabeled data within a variational framework allows the model to capture rich semantic patterns. Instead of permanently deleting sentences due to initial classification errors, the model gradually uncovers their true underlying relations.
By transitioning data from a state of harmful noise to a state of helpful unlabeling, and finally to rectified cleanliness, these frameworks maximize data efficiency. As Hui Kang et al. show in Unleashing the Potential of Regularization Strategies in Learning with Noisy Labels, leveraging low-confidence samples as unlabeled data allows networks to survive extreme environments where over half of the initial annotations are completely wrong, turning a data-poisoning disaster into a successful semi-supervised training run.