Imagine you are training a machine learning model to distinguish between wolves and huskies. In a standard, simplified noise model, we assume that an annotator flips a label with a fixed probability based purely on the true class—say, 10% of all wolf images are mistakenly labeled as huskies, regardless of what the individual images look like. Under this assumption, a clear, high-resolution photo of a timber wolf howling in the snow is just as likely to be mislabeled as a blurry, ambiguous photo of a dog taken from a distance.
In the real world, human errors are rarely random. An annotator won't mislabel a textbook image of a wolf. Instead, they make mistakes on ambiguous, low-quality, or atypical instances—such as a white husky in a blizzard or a wolf-dog hybrid. This is instance-dependent label noise (also called feature-dependent noise). The probability of label corruption is a function of the input features $x$, not just the true label $y$.
Why Class-Conditional Models Fail
Early noise-robust methods, such as the loss correction techniques proposed by Giorgio Patrini et al. in Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach (and earlier by Alessandro Rozza et al. in Making Neural Networks Robust to Label Noise: a Loss Correction Approach), rely on estimating a global noise transition matrix $T$. Here, $T_{ij} = P(\tilde{y}=j \mid y=i)$ represents the probability that true class $i$ is flipped to noisy label $j$. Similarly, Sainbayar Sukhbaatar et al. in Training Convolutional Networks with Noisy Labels introduced a constrained linear layer on top of the standard softmax to learn this global class-level transition matrix.
While elegant, a global matrix assumes every instance within a category shares the exact same error rate. When noise is instance-dependent, this assumption breaks down completely. As Pengfei Chen et al. demonstrate in Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise, instance-dependent noise causes the clean and noisy subsets of a dataset to occupy entirely different regions of the feature space. If a model tries to apply a uniform correction or a global filtering threshold, it suffers from a "support mismatch," discarding difficult but clean examples or failing to filter out structured errors.
Core Strategies for Modeling Instance-Dependent Noise
To handle noise that changes from image to image, researchers have developed mechanisms that move away from global dataset properties toward granular, context-aware models.
1. Image-Conditional and Variational Noise Modeling
Instead of a static matrix, we can treat the label transition probability as a function of the input features, $T(x)$. An early instance of this was proposed by Tong Xiao et al. in Learning from massive noisy labeled data for image classification, using a probabilistic graphical model to determine whether a label is correct, visually confusing, or random based on specific image features.
In the context of partial label learning, Ning Xu et al. developed VALEN in Instance-Dependent Partial Label Learning. VALEN uses variational inference to iteratively reconstruct the latent, true label distribution for each specific instance. By recognizing that incorrect candidate labels are often semantically or visually related to the ground truth, the framework models the candidate label generation as an instance-dependent process rather than uniform random noise.
2. Confidence Regularization and Sample Sieves
Another approach avoids explicitly estimating $T(x)$ by dynamically filtering or weighting samples based on how the model's confidence evolves during training. Deep neural networks exhibit a "memorization effect," where they learn clean, simple patterns before memorizing noisy labels. However, because instance-dependent noise targets complex or ambiguous samples, distinguishing between an inherently difficult clean sample and a corrupted sample is exceptionally challenging.
To solve this, Hao Cheng et al. proposed CORES² (Confidence Regularized Sample Sieve) in Learning with Instance-Dependent Label Noise: A Sample Sieve Approach. CORES² uses a confidence-regularized loss function to dynamically separate corrupted examples from clean ones without needing a pre-specified noise rate. By monitoring the model's prediction confidence on individual samples over time, the sieve filters out instances where the label conflicts with the structured feature representations learned by the network.
3. Self-Evolution and Temporal Label Refinement
Rather than discarding samples, we can iteratively correct the labels themselves. In Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise, the authors introduced SEAL (Self-Evolution Average Label). SEAL leverages the model's own historical predictions to refine the training targets. By averaging a model's softmax outputs over multiple epochs, the algorithm smooths out the noise on an instance-by-instance basis, allowing the network to gradually correct its own supervision signal for ambiguous inputs.
Theoretical Guarantees and Downstream Impacts
Can we guarantee that a model will converge to the optimal classifier when the noise depends on the features? Aditya Krishna Menon et al. proved in Learning from Binary Labels with Instance-Dependent Corruption that optimal classification and ranking are still achievable under instance-dependent corruption, demonstrating that algorithms like Isotron can efficiently learn without explicit knowledge of the underlying noise process.
However, ignoring instance-dependent noise has severe consequences beyond accuracy. In Fair Classification with Instance-dependent Label Noise, Bo Han demonstrated that when label noise correlates with specific instance features, standard fairness-aware algorithms can actually become more biased. By explicitly modeling the instance-dependent noise process using a causal variational autoencoder framework, the author showed it is possible to simultaneously enforce counterfactual fairness and maintain robustness against structured label corruption.