When training deep neural networks on real-world datasets, incorrect or "noisy" labels present a major challenge. If a network is left to train blindly on corrupted data, it will eventually memorize the errors, degrading its ability to generalize.
To prevent this, we need a way to separate clean samples from noisy ones. Rather than relying on manual thresholds or expensive clean validation sets, a powerful class of techniques models the network's own internal behavior using statistical mixture models. This approach turns a deep learning liability—the tendency to fit data over time—into an unsupervised diagnostic tool.
The Intuition: The Memorization Effect
The core reason this strategy works lies in a phenomenon known as the memorization effect. During the early stages of training, deep neural networks do not learn all data points at the same rate. Instead, they prioritize learning simple, consistent, and clean patterns first. Only much later in the training cycle do they begin to overfit and memorize complex, inconsistent noise.
As a result of this staggered learning process, a stark divergence occurs in the network's per-sample loss values:
- Clean samples quickly align with the true underlying patterns of the data, causing their training loss to drop rapidly.
- Noisy samples contradict these emerging patterns, causing their training loss to remain high during early and middle training phases.
As noted by Bing Han et al. in Self-Supervised Speaker Verification Using Dynamic Loss-Gate and Label Correction, this behavioral split causes the training loss across the dataset to naturally form a bimodal distribution. If you plot a histogram of the losses, you will see two distinct peaks: a low-loss peak representing clean data, and a high-loss peak representing contaminated data.
Statistical Partitioning via Mixture Models
Once we recognize that the loss space is split into two populations, the challenge becomes finding the optimal boundary between them. Because the exact shape of this distribution shifts depending on the dataset, the architecture, and the noise rate, a fixed, hard-coded threshold is impractical.
Instead, researchers fit a mixture model to the loss distribution in an unsupervised, online manner. In the foundational framework introduced by Junnan Li et al. in DivideMix: Learning with Noisy Labels as Semi-supervised Learning, a two-component Gaussian Mixture Model (GMM) is applied to the per-sample cross-entropy losses.
Using the Expectation-Maximization algorithm, the GMM automatically finds the means and variances of the two underlying distributions:
- A low-loss component representing high-confidence, clean samples.
- A high-loss component capturing low-confidence, noisy samples.
While GMMs are highly effective, the loss values are often bounded and highly skewed. To accommodate this specific geometry, Eric Arazo et al. demonstrate in Unsupervised Label Noise Modeling and Loss Correction that a Beta Mixture Model (BMM) can be used instead. Because the Beta distribution is bounded between 0 and 1, it can more precisely fit the asymmetric, heavily skewed distributions typical of normalized neural network losses.
From Binary Filtering to Soft Probabilities
The primary advantage of using a mixture model over a hard threshold is that it outputs a continuous posterior probability of cleanliness for every single sample. This soft metric allows training frameworks to handle data much more flexibly than simply throwing "bad" samples away.
Semi-Supervised Learning
Instead of discarding high-loss samples—which might still contain valuable visual or textual features—frameworks like the one proposed by Junnan Li et al. in DivideMix split the dataset into a labeled set (high clean probability) and an unlabeled set (low clean probability). The system then trains using semi-supervised techniques, leveraging the features of the noisy data without trusting its corrupted labels.
Soft Label Refurbishment
Rather than making a binary choice between the original label and the model's prediction, the clean probability can act as a dynamic blending weight. MingCai Chen et al. utilize these scores in Robust Logit Adjustment for Learning with Long-Tailed Noisy Data to smoothly refurbish labels, relying more on the model's internal predictions when the mixture model flags a sample as highly likely to be noisy.
Multi-Component Isolation
Real-world noise is rarely uniform. In environments containing both closed-set noise (mislabeled within known classes) and open-set noise (entirely out-of-distribution samples), a simple clean/noisy split is insufficient. Ragav Sachdeva et al. show in EvidentialMix: Learning with Combined Open-set and Closed-set Noisy Labels that expanding the framework to a three-component GMM can successfully isolate clean data, closed-set noise, and open-set noise into three distinct statistical clusters based on their varying degrees of uncertainty.
Beyond Classification Loss
The intuition of mixture-based partitioning generalizes far beyond standard image classification. Any training metric that exhibits a bimodal divergence between clean and corrupted pairs can be modeled this way.
For instance, in cross-modal applications like image-text retrieval, Zihua Zhao et al. apply a GMM to cross-modal similarity indicators in Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning to automatically partition clean and mismatched pairs.
Furthermore, when models attempt to self-correct by generating their own labels, they risk falling into a confirmation bias loop where they reinforce their own mistakes. To break this loop, Paul Albert et al. introduce a method in Is your noise correction noisy? PLS: Robustness to label noise with two stage detection that fits a GMM to the distribution of pseudo-losses—the loss calculated against the model's own generated pseudo-labels. This allows the system to statistically separate reliable, high-confidence corrections from under-confident, erroneous guesses before they can corrupt the model.