When training deep neural networks, we typically assume that while our datasets might be imperfect, the correct labels still form the clear majority. This assumption underpins the classic "small-loss trick": because neural networks tend to learn clean, consistent patterns before memorizing random noise, samples that yield low training losses early on can be safely treated as clean.
But what happens when this assumption is completely inverted? In extreme noise regimes—where 60%, 80%, or even 98% of the labels are corrupted—the clean data becomes a minority signal buried under a mountain of structured or random errors.
In this extreme environment, traditional noise-mitigation strategies do not just degrade; they catastrophically collapse. To train models successfully when the vast majority of labels are wrong, researchers have had to move away from local, instance-level filtering and instead rely on global distribution modeling, label-agnostic representation learning, and external semantic anchors.
The Self-Confirmation Trap of Local Denoising
To understand why extreme label noise requires a fundamental paradigm shift, we must first look at why standard denoising techniques fail. Many traditional algorithms rely on local neighborhood consistency or instance-level confidence scores to identify and fix corrupted labels. The intuition is straightforward: if a sample's label disagrees with the labels or predictions of its closest geometric neighbors in the feature space, it is likely noisy.
Low-Noise Regime (<20% Noise) Extreme-Noise Regime (>70% Noise)
[Clean] [Noisy]
/ | \ / | \
[Clean] [Clean] [Noisy] [Noisy] [Noisy] [Clean]
(Majority consensus (Majority consensus
corrects the noise) reinforces the error)
As highlighted by Yuanhong Chen et al. in BoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classification, this logic completely breaks down once the noise ratio crosses a critical threshold (typically around 50% to 60%). When 80% of the labels are incorrect, a sample's neighbors are also highly likely to be incorrectly labeled.
If a model attempts to correct a label by aggregating information from its neighborhood, it inadvertently pulls in corrupted signals. Zhizhong Huang et al. demonstrate in Twin Contrastive Learning with Noisy Labels that local consensus mechanisms end up confirming and reinforcing errors rather than identifying them as noise. This is known as self-confirmation bias.
Similarly, Wenkai Zhang et al. observe in Denoising Distantly Supervised Named Entity Recognition via a Hypergeometric Probabilistic Model that instance-level confidence scores become fundamentally corrupted in high-noise environments. Because the model's internal representations are trained on predominantly false data, its own confidence metrics become a reflection of the noise distribution rather than the true underlying categories.
Strategy 1: Label-Agnostic Representation Learning
If the labels are mostly garbage, the model cannot rely on them to build its initial understanding of the data. The solution is to decouple feature learning from the noisy labels entirely.
Instead of forcing a network to map an image directly to a potentially corrupted category, frameworks designed for extreme noise leverage contrastive learning. By maximizing the similarity between different augmented views of the same instance (and minimizing similarity between different instances), the network learns to group semantically similar data together purely based on intrinsic visual or structural features.
Step 1: Contrastive Learning (Label-Agnostic)
[Image A1] <---> [Image A2] (Group by visual similarity, ignoring labels)
Step 2: Global Label Assignment
[ Visual Cluster A ] <=== Map to ===> [ Global Label Distribution ]
Nazmul Karim et al. utilize this mechanism in UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning to maintain robust feature representations even when up to 98% of the training labels are wrong. Because the contrastive objective operates independently of the label space, the model's feature extractor remains uncorrupted by the pervasive label errors. Once a clean, label-agnostic feature space is established, separating the true signal from the noise becomes a much simpler downstream task.
Strategy 2: Global Distribution Modeling and Semi-Supervised Reframing
Once a robust feature space is established, how do we actually utilize the noisy labels without falling back into the self-confirmation trap? Rather than making hard, instance-by-instance decisions about whether a label is "clean" or "noisy," modern approaches model the loss or label behavior globally across the entire dataset.
A foundational example of this is DivideMix, developed by Junnan Li et al., which reframes the noisy label problem as a semi-supervised learning (SSL) task. Instead of trying to correct every label, the framework uses a Gaussian Mixture Model (GMM) to dynamically model the historical loss distribution of all samples.
Frequency
| ____ ________
| / \ / \
| / Clean\ / Noisy \
|___/________\_______/____________\___
0 High Early Loss
Even in extreme noise scenarios, clean samples still exhibit a marginally more consistent loss profile than completely randomized labels. By fitting a global mixture model to these losses, the authors divide the dataset into a small, highly confident "labeled" set (the clean minority) and a massive "unlabeled" set (the noisy majority). The problem is then handed over to powerful semi-supervised algorithms that propagate information from the trusted minority to the rest of the dataset.
To make these global models more stable when the signal-to-noise ratio is incredibly low, researchers often shift from hard categorical assignments to continuous distributions. For instance, Eric Arazo et al. show in Unsupervised Label Noise Modeling and Loss Correction that modeling training losses with a Beta Mixture Model allows the network to dynamically scale its learning updates.
In a similar vein, the generative framework proposed by Kuan-An Su and Yan Su in Let the noise flow away: combating noisy labels using normalizing flows uses conditional normalizing flows to map the entire label distribution. By allowing the network to absorb label uncertainty into a continuous probability flow rather than forcing a hard classification, the model avoids memorizing extreme, high-ratio corruptions.
Strategy 3: Anchoring to External Truth
When the noise ratio approaches 90% or higher, relying entirely on internal dataset statistics becomes incredibly risky. If a model has no uncorrupted baseline, it can easily mistake a highly structured noise pattern for the true underlying data distribution. To prevent this "normality pollution," the latest generation of algorithms introduces external anchors.
One approach is to use a tiny, pristine validation set to guide the learning process. As explored by Murong Ma et al. in TrainRef: Curating Data with Label Distribution and Minimal Reference for Accurate Prediction and Reliable Confidence, a minimal extrinsic reference set acts as a ground-truth anchor. It provides a baseline distribution that prevents the model's internal meta-optimizers from drifting toward the prevalent noise patterns.
Alternatively, instead of requiring manual curation of a clean reference set, researchers have begun leveraging pre-trained foundation models as impartial judges.
[Noisy Dataset Image] ──────> [Pre-trained CLIP Model]
│
(Zero-shot Semantic Check)
▼
[Impartial Clean/Noisy Decision]
In CLIPCleaner: Cleaning Noisy Labels with CLIP, Georgios Tzimiropoulos demonstrates that by decoupling sample selection from the active training loop and using the zero-shot semantic capabilities of a vision-language model, you can bypass internal dataset bias entirely. Because the foundation model's representations were frozen prior to exposing it to the noisy training set, it remains completely immune to the high-ratio label corruption, providing a reliable compass to kickstart the denoising process.
Key Foundational Work
t-logistic regression (2010) — contributed 1 concept elements to this direction.
Unsupervised One-Class Learning for Automatic Outlier Removal (2014) — contributed 1 concept elements to this direction.