Learning with Instance Dependent Label Noise

This direction is merged into another direction: instance-dependent label noise. Its last version stays readable and citable here.

Version 1 of 111 papers16 conceptsfrozen 2026-10-04

In real-world datasets, labels are rarely perfect. A common way to handle this is to assume class-conditional noise, where the probability of a "cat" being mislabeled as a "dog" is a fixed value across the entire dataset. However, this assumption fails in practice. A blurry, low-contrast photo of a cat is far more likely to be mislabeled than a high-resolution, centered portrait.

This shift—from noise that depends on the category to noise that depends on the specific image—is known as instance-dependent label noise. Recent research has moved toward modeling this complexity by treating the "noise" not as a global constant, but as a function of the input features themselves.

The Problem of Identifiability

Mathematically, we model label noise using a transition matrix $T$, which describes the probability of a true label $y$ flipping to a noisy label $\tilde{y}$. In the class-conditional setting, $T$ is a simple $K \times K$ matrix (where $K$ is the number of classes).

In the instance-dependent setting, the matrix becomes $T(x)$, meaning every single data point $x$ has its own unique transition matrix. This creates a massive "identifiability" problem: if every image has its own unique error pattern, how can a model possibly distinguish between the true underlying pattern and the noise? Without constraints, the model could simply "explain away" any label by claiming that specific image has a 100% chance of being mislabeled.

Parameterizing the Noise Function

To make this problem tractable, researchers like Shuo Yang, Erkun Yang, and Gang Niu propose using a dedicated neural network to estimate $T(x)$. In Estimating Instance-dependent Label-noise Transition Matrix using DNNs, they argue that instead of trying to map noisy labels back to "clean" labels (which are unobservable), we should map them to Bayes optimal labels.

Because Bayes labels are deterministic functions of the input distribution, they are easier for a neural network to approximate. By training a parametric model to predict the transition from these optimal labels to the observed noisy ones, the system can generalize noise patterns from simple, clear examples to more ambiguous ones.

Anchoring Noise with Confidence Scores

Another breakthrough involves using "side information" to anchor the noise estimation. Antonin Berthon, Bo Han, and Tongliang Liu demonstrate in Confidence Scores Make Instance-dependent Label-noise Learning Possible that if we have a "confidence score" for a label—perhaps from a human annotator or a pre-trained teacher model—the high-dimensional problem of estimating $T(x)$ becomes solvable.

These confidence scores effectively populate the diagonal of the transition matrix (the probability that the label is correct). By fixing this diagonal, the model only needs to learn how the remaining "error probability" is distributed among the other classes. This approach, known as Instance-Level Forward Correction (ILFC), allows models to recover clean-label performance even when a significant portion of the dataset is corrupted.

Reducing Complexity via Embeddings

If estimating a full $K \times K$ matrix for every instance is too computationally expensive, we can simplify the noise model. Masashi Sugiyama and Yivan Zhang introduced Instance-Confidence Embedding (ICE) in Approximating Instance-Dependent Noise via Instance-Confidence Embedding.

Instead of a full matrix, ICE approximates the noise with a single trainable scalar for each data point. This scalar represents the "trustworthiness" of that specific instance. This reduction prevents the parameter explosion that usually occurs with many classes, providing a scalable way to identify ambiguous or mislabeled samples in large-scale datasets.

Intuition from Similarity

Finally, we can use the structure of the data itself to regularize the noise. In Instance-Dependent Label-Noise Learning with Manifold-Regularized Transition Matrix Estimation, De Cheng and Nannan Wang apply a simple human intuition: similar-looking images should have similar probabilities of being mislabeled.

By applying manifold regularization, the model ensures that the transition matrices $T(x)$ vary smoothly across the feature space. If the model learns that one specific image of a "wolf" is likely to be mislabeled as a "husky," the regularization forces it to assume the same for visually similar images. This constraint stabilizes the learning process, preventing the model from over-fitting to individual label errors.

Beyond Supervised Learning

These concepts are also finding a home in semi-supervised learning. Muyang Li and Xun Yang developed InstanT (Semi-supervised Learning with Instance-dependent Thresholds), which uses an auxiliary head to predict the likelihood of pseudo-label errors. Instead of using a flat confidence threshold for all unlabeled data, InstanT calculates a unique threshold for every sample based on its specific ambiguity, ensuring that only the most reliable pseudo-labels are used for training.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_learning_with_instance_dependent_label_n_3c0e2eae,
  title = {Learning with Instance Dependent Label Noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-with-instance-dependent-label-noise/rdr_3c0e2eae219343f2a0b08365ab7ff9ed},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_with_instance_dependent_label_n_3c0e2eae_v1,
  title = {Learning with Instance Dependent Label Noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-with-instance-dependent-label-noise/rdr_3c0e2eae219343f2a0b08365ab7ff9ed/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}