Learning with Instance-Dependent Label Noise

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2559 papers848 conceptsfrozen 2026-10-04

In the idealized world of machine learning research, label noise is often treated as a coin flip. We assume that if a "cat" is mislabeled, it is because of a random process that affects all cats equally. This is known as Class-Conditional Noise (CCN).

However, real-world data is rarely so uniform. As Tong Xiao, Xiaogang Wang, and colleagues argued in Learning from massive noisy labeled data for image classification, label reliability is actually a function of the specific image features. A blurry, low-resolution, or poorly lit image of a cat is far more likely to be mislabeled by a human or an automated tagger than a clear, prototypical one. This is Instance-Dependent Noise (IDN), and it represents a fundamental shift in how we must approach robust learning.

The Memorization Trap

Deep neural networks possess a "mixed blessing": they tend to learn simple, generalizable patterns before they begin to memorize noise. In a CCN setting, where noise is random, the network sees the noise as high-frequency "static" that is difficult to fit, giving researchers a window to stop training early.

IDN breaks this defense. Because the noise is correlated with the features of the instance—such as visual ambiguity or semantic similarity between classes—the network perceives the noise as a complex but legitimate pattern. Jiaheng Wei, Zhaowei Zhu, and their team demonstrated in Learning with Noisy Labels Revisited that models memorize these feature-correlated human mistakes much faster than synthetic random noise. When the error "makes sense" visually, the model is more likely to trust it, leading to a biased classifier that fails to generalize to the true distribution.

Moving Beyond Class-Level Probabilities

To combat IDN, we must move from global noise rates to instance-level reliability. The research direction has converged on three primary strategies:

1. Dynamic Sample Sieving

Instead of trusting all labels, we can treat the training process as a filter. In Learning with Instance-Dependent Label Noise: A Sample Sieve Approach, Zhaowei Zhu and Yang Liu introduced CORES$^2$. This framework uses confidence regularization to "sieve" out corrupted labels dynamically. By monitoring how the model's confidence in a label evolves, the system identifies samples where the label contradicts the emerging feature patterns, effectively discarding the noise without needing to know the noise rate in advance.

2. Self-Evolution and Consensus

If a model is trained on noisy data, its own predictions over time might actually be more reliable than the original labels. Pengfei Chen and colleagues proposed SEAL (Self-Evolution Average Label) in Beyond Class-Conditional Assumption. SEAL corrects noisy labels by averaging a model's predictions across multiple epochs. The intuition is that while a model might momentarily memorize a "hard" noisy label, its aggregate internal representation across the training process will eventually favor the more consistent, underlying structure of the data.

3. Noise-Invariant Loss Functions

Another approach is to design loss functions that are mathematically "blind" to noise. Yang Liu and Hongyi Guo developed Peer Loss Functions (Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates). By comparing a model's prediction on a sample to its prediction on a random "peer" sample, the loss function can cancel out the bias introduced by label noise. This creates a training signal that remains stable even when the relationship between features and labels is corrupted.

The Shift to Realistic Benchmarks

A recurring theme across this research—highlighted by DeepMind researchers in A Realistic Simulation Framework for Learning with Label Noise—is that our benchmarks must reflect this complexity. Traditional evaluations using uniform label flipping provide a false sense of security.

True progress in the field now relies on datasets like CIFAR-10N and Clothing1M, which capture the systematic confusion humans experience when faced with ambiguous inputs. By treating noise as a function of the instance ($x$) rather than just the class ($y$), these researchers are building models that can finally navigate the "messy" reality of human-annotated data.

Papers in this version

Showing 60 of 559 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 471 papers
  2. Version 1 · 2026-10-04 · 559 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_instance_dependent_label_noise_46689a3d,
  title = {instance-dependent label noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/instance-dependent-label-noise/rdr_46689a3d2e5148e5aeee550f43f2c1f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_with_instance_dependent_label_n_46689a3d_v1,
  title = {Learning with Instance-Dependent Label Noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-with-instance-dependent-label-noise/rdr_46689a3d2e5148e5aeee550f43f2c1f0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}