Deep neural networks possess a remarkable but double-edged capability: they can fit virtually any training distribution, including completely random labels. When training on real-world datasets scraped from the web or annotated via low-cost crowdsourcing, this capacity becomes a liability. Networks exhibit a predictable "memorization effect"—they first learn clean, generalized patterns from consistent data before slowly adapting their parameters to memorize individual, erroneous labels.
To prevent this memorization, traditional noise-robust frameworks often rely on heavy algorithmic machinery. Strategies like "Co-teaching" or teacher-student frameworks require training multiple deep networks simultaneously so they can cross-validate each other's samples. Other approaches deploy complex multi-stage pipelines or auxiliary networks to estimate noise transition matrices. While effective, these methods double the computational footprint, increase memory overhead, and complicate standard training workflows.
A prominent research direction focuses on eliminating this computational overhead. By modifying the loss landscape, tracking internal model dynamics, or leveraging data-centric filtering, researchers are developing lightweight, single-network mechanisms that achieve state-of-the-art noise tolerance with virtually zero added training cost.
Reshaping the Loss Landscape
The most direct way to make a single network inherently robust is to replace the standard Categorical Cross Entropy (CE) loss. To understand why CE fails under label noise, consider its mathematical behavior: it penalizes incorrect predictions with an unbounded logarithmic scale. If an image of a cat is mistakenly labeled as a dog, a well-trained model will output a low probability for "dog." CE will heavily penalize this correct intuition, forcing the network to distort its decision boundaries to satisfy the erroneous label.
To counteract this, Ghosh et al. demonstrated that mathematically "symmetric" loss functions are inherently robust to label noise. A loss function is symmetric if the total loss summed over all possible classes for a given sample remains constant. They proved that Mean Absolute Error (MAE) satisfies this condition; under symmetric noise, minimizing MAE on a noisy dataset yields the same global minimizer as minimizing it on a clean dataset.
However, MAE can suffer from slow convergence and underfitting on complex datasets. To balance learning speed and noise tolerance, Ma et al. proposed Symmetric Cross Entropy (SL). They coupled CE with a noise-robust "Reverse Cross Entropy" (RCE) term:
$$\mathcal{L}_{\text{SL}} = \alpha \mathcal{L}_{\text{CE}} + \beta \mathcal{L}_{\text{RCE}}$$
While CE evaluates the model's prediction given a hard target, RCE evaluates the true target given the model's prediction. This simple modification prevents the unbounded penalty on noisy labels while maintaining the rapid convergence of standard cross-entropy.
Taking this plug-and-play philosophy further, Wang et al. introduced $\epsilon$-Softmax, a two-line code replacement for the standard softmax layer. By forcing network outputs to approximate one-hot vectors, it naturally mitigates the gradient updates driven by corrupted labels without requiring multi-stage sample selection or architectural overhead.
Exploiting Internal Model Dynamics
Instead of training a second network to identify clean samples, a single network can look inward. Because networks learn clean patterns before memorizing noise, their own internal signals—such as prediction trajectories and probability distributions—betray which labels are suspicious.
For example, Zhao et al. introduced P-DIFF, a paradigm that monitors the "probability difference" distribution. In a clean sample, a maturing network quickly develops a massive gap between the predicted probability of the assigned label and the next most likely class. For a mislabeled sample, the network experiences internal conflict: its feature representations align with the true semantic class, but the loss function forces it toward the noisy label. This creates a distinct, narrow probability difference. By tracking this single metric, the authors automatically down-weight suspicious samples during standard training cycles.
Clean Sample: [True Class: Cat] Prediction: Cat (0.92) -> Runner-up: Dog (0.03) ==> Large Gap
Mislabeled Sample: [Noisy Label: Dog] Prediction: Cat (0.65) -> Target: Dog (0.20) ==> Narrow/Conflicted Gap
Similarly, Song developed MORPH, a strategy that automatically identifies the exact "transition point" where a network finishes its general learning phase and begins memorizing noise. Rather than relying on fixed epoch hyperparameters, MORPH monitors the model's internal state to switch from a global training phase to a selective evolution phase, cutting training time by up to 3x compared to multi-network filtering methods. In the context of multi-label classification, Kim et al. showed that samples maintaining an unusually large loss late in the training cycle are precisely the ones the model is struggling to memorize, allowing a lightweight filtering mechanism to isolate and neutralize them.
Data-Centric and Training-Free Filtering
Another way to bypass the overhead of multi-network co-training is to decouple the label-cleaning process from the model's optimization loop entirely. If corrupted labels can be identified before or alongside training using lightweight heuristics, the network can be trained normally.
Zhu et al. introduced SimiFeat, a completely "training-free" approach to label cleaning. Instead of optimizing a complex model to predict label correctness, SimiFeat leverages the geometric properties of the data's feature space. By analyzing the neighborhood information of data features, the authors demonstrate that label errors can be accurately detected based on semantic distance alone, removing the need for resource-intensive meta-learning loops.
When external models are used, they can be leveraged as frozen, impartial judges rather than active training components. Feng introduced CLIPCleaner, which uses the pre-trained, frozen vision-language model CLIP to evaluate dataset labels against image semantics before training begins. Because the cleaning mechanism is static and decoupled, it completely avoids the "self-confirmation bias" that occurs when a model attempts to clean the same data it is learning from.
For scenarios where online batch selection is necessary, Lisicki et al. reframed sample selection as a combinatorial bandit problem. By using the Follow the Perturbed Leader (FPL) algorithm, the training pipeline dynamically selects the most informative, likely clean samples for each batch without requiring auxiliary neural networks or doubling hardware resources.
Regularization as an Inductive Bias
A final, elegant realization within this research direction is that standard architectural constraints can act as a natural barrier against noise memorization. Memorizing random labels requires a network to learn highly complex, high-frequency decision boundaries. By restricting the network's capacity or information flow, we can introduce an inductive bias that favors simple, clean patterns.
Chen et al. achieved this by introducing a "compression inductive bias." By strategically applying Dropout and Nested Dropout, the authors limited the network's information capacity. Because clean patterns are shared across many images, they easily pass through a compressed bottleneck. Conversely, the unique, idiosyncratic pixel patterns required to memorize a corrupted label are filtered out.
Ultimately, Kang et al. demonstrated that the complex, multi-stage algorithms dominating the literature can often be outperformed by a simple baseline. By pairing standard cross-entropy loss with a strict recipe of standard regularization strategies—specifically sharp learning rate decay and model weight averaging—a single network can resist noise memorization and generalize effectively, proving that robust learning does not inherently require computational complexity.
Key Foundational Work
Decoupling "when to update" from "how to update" (2017) — contributed 1 concept elements to this direction.
Symmetric Cross Entropy for Robust Learning With Noisy Labels (2019) — contributed 1 concept elements to this direction.
Robust Loss Functions under Label Noise for Deep Neural Networks (2017) — contributed 1 concept elements to this direction.