Learning from Noisy Labels Without Clean Supervision

69 concepts69 papers1 proposal

Deep neural networks possess a notorious capacity to memorize. When trained on datasets containing corrupted or incorrect labels—a common byproduct of automated web scraping or hurried crowdsourcing—a standard network will initially capture the true underlying signal but eventually overfit to the errors.

To prevent this memorization, early noise-robust algorithms relied on two critical crutches: a small, human-verified clean validation set to guide the model, or an exact prior estimate of the noise rate to tune filtering thresholds. However, as noted by Yang et al. and Han et al., requiring a "gold standard" clean subset creates a severe scalability bottleneck for massive datasets. Similarly, Sastry et al. and Shen et al. point out that the true noise proportion is almost always unknown in real-world deployments.

A rapidly growing body of research focuses on breaking these dependencies. By eliminating the need for clean reference data or predefined noise statistics, these methods enable models to autonomously estimate noise distributions and learn robustly from entirely uncurated, "dirty" data.

Exploiting Learning Dynamics

How can a model distinguish a correct label from an incorrect one without any clean reference? The answer lies in the temporal behavior of neural networks during training, a phenomenon known as early learning.

During the initial phases of training, networks naturally learn simple, generalized patterns (the clean signal) before expanding their capacity to memorize complex, chaotic patterns (the noise). This creates a distinct divergence in the loss distribution: clean samples yield low losses early on, while noisy samples maintain high losses.

Instead of using a fixed threshold based on a known noise rate to separate these samples, recent methods model this loss distribution dynamically. For instance, Arazo et al. apply an unsupervised Beta Mixture Model directly to training losses to estimate the probability that any given sample is mislabeled. Similarly, Yang et al. introduce Dynamic Loss Thresholding (DLT) to adaptively separate clean and noisy samples as training progresses, treating suspected noisy samples as unlabeled data in a semi-supervised framework.

[Training Loss Distribution]
Early Training:
Clean Samples (Low Loss)  --->  ■ ■ ■ ■ ■
Noisy Samples (High Loss) --->              ■ ■ ■ ■ ■
                                    ^
                        Dynamic Threshold Estimated 
                        Without Prior Noise Rate

Beyond raw loss values, the geometric properties of a model's internal representations change during this transition. Ma et al. track this shift using Local Intrinsic Dimensionality (LID), observing that clean data lies on low-dimensional manifolds while noisy data forces the network into high-dimensional subspaces. This geometric insight is extended by Zhang et al. in their CoLafier framework, which uses LID to guide a collaborative discriminator without requiring prior noise statistics.

Grounding Labels in Intrinsic Data Geometry

When labels are completely untrustworthy, a powerful alternative is to anchor the learning process in the self-contained structure of the data itself. If a label contradicts the visual or semantic similarity of its neighbors, it is likely corrupt.

Zhu et al. demonstrate this data-centric shift with SimiFeat, a training-free approach that identifies corrupted labels purely by analyzing the feature neighborhoods of data points, removing the need for a clean validation set to set filtering thresholds.

Another strategy blends supervised classification with self-supervised objectives. Because self-supervised learning relies entirely on data augmentations rather than human labels, its representations are inherently immune to label noise. Tan et al. introduce Co-learning, which pairs a standard classifier with a self-supervised contrastive learning head on a shared encoder. By forcing the network to respect the intrinsic visual structure of the data, the model resists blindly memorizing incorrect labels, achieving robustness without any prior knowledge of the noise rate.

End-to-End Noise Modeling and Loss Decomposition

Rather than explicitly filtering out noisy samples, another branch of this research integrates the noise distribution directly into the model architecture or the mathematical formulation of the loss function.

An early foundational approach introduced a dedicated "noise layer" on top of the standard softmax output to automatically learn the label transition matrix directly from dirty data without clean supervision (Training Convolutional Networks with Noisy Labels). This concept was later adapted to text classification by Jindal et al., who utilized a non-linear noise layer to absorb label errors during training.

Alternatively, the problem can be framed as a statistical decomposition. Ding et al. propose MC-LDCE, a framework that mathematically decomposes the loss function and estimates class centroids to provide an unbiased training objective, eliminating the need for perfectly labeled auxiliary data. On the theoretical front, Clayton Scott established convergence rates for Mixture Proportion Estimation (MPE), proving that classification with completely unknown label noise can be made statistically consistent under mild distributional assumptions.

Key Foundational Work

Classifier Learning from Noisy Data as Probabilistic Evidence Combination (1992) — contributed 1 concept elements to this direction.
Imbalanced Multiple Noisy Labeling for Supervised Learning (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_clean_data_dependency_unknown_noise_dist_4900e99b,
  title = {clean data dependency (unknown noise distribution)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/clean-data-dependency-unknown-noise-distribution/rdr_4900e99ba3f9498fb65d3af07215774c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_clean_data_dependency_unknown_noise_dist_4900e99b_v1,
  title = {clean data dependency (unknown noise distribution)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/clean-data-dependency-unknown-noise-distribution/rdr_4900e99ba3f9498fb65d3af07215774c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake