When training deep neural networks on real-world datasets, practitioners frequently encounter corrupted or "noisy" labels. Standard training objectives like Categorical Cross Entropy (CCE) are notoriously fragile in these environments. Because CCE heavily penalizes large errors, it forces the network to aggressively alter its weights to fit every sample—including the mislabeled ones. This behavior causes the network to "memorize" the noise, destroying its ability to generalize to clean data.
To combat this, researchers introduced noise-robust loss functions. A landmark finding by Ghosh et al. proved that "symmetric" loss functions, such as Mean Absolute Error (MAE), are theoretically immune to label noise; their global minima remain unchanged whether the data is clean or corrupted.
Yet, in practice, substituting CCE with MAE often causes a severe drop in final accuracy. While robust losses successfully prevent the model from overfitting to noise, they introduce a secondary, equally crippling problem: severe underfitting. This creates a fundamental optimization-robustness trade-off. Objectives that are easy to optimize tend to overfit noise, while objectives that resist noise often fail to converge on complex datasets.
The Mechanics of Gradient Saturation
To understand why robust losses underfit, we must look at the dynamics of their gradients.
Consider how a network learns from an ambiguous or highly incorrect sample under CCE versus MAE. In CCE, the gradient magnitude is proportional to the model's error. If the network is highly uncertain or completely wrong about a sample, it receives a massive gradient signal forcing it to learn.
In contrast, the gradient of MAE has a constant magnitude regardless of the prediction error. This constant gradient behavior introduces two major optimization bottlenecks:
- Gradient Saturation: As observed by Ishiguro et al., when a model faces complex, high-dimensional datasets, flat or constant gradients fail to provide a sufficient learning signal during the early phases of training. The optimization process simply stalls.
- Equal Sample Weighting: Because the gradient magnitude is uniform, MAE treats every sample identically. As Ye et al. point out, this prevents the model from prioritizing informative, hard-but-clean samples over easy ones, leading to slow convergence and a lack of "learning sufficiency."
Furthermore, Ma et al. demonstrated that robust losses frequently exhibit flat gradients when the model is highly uncertain. The very mathematical constraints required to ignore outliers prevent the model from fully optimizing on clean, informative samples (Zhou et al.; Ahn et al.). The model becomes so restricted in its attempt to avoid noise that it fails to extract the intricate features necessary for high-performance classification (Li et al.; Lin et al.).
Bridging the Gap via Loss Interpolation
To resolve this bottleneck, one prominent research direction focuses on constructing mathematical bridges that smoothly interpolate between the rapid convergence of CCE and the noise tolerance of MAE.
Cross Entropy (CCE) <-------------------> Mean Absolute Error (MAE)
[Fast / Overfits Noise] Tunable Space [Slow / Underfits Clean Data]
Sabuncu et al. introduced the $L_q$ loss, which uses a hyperparameter $q$ to span this spectrum. When $q \to 0$, the loss behaves like CCE; when $q = 1$, it becomes MAE. By tuning $q$, practitioners can inject enough gradient signal to drive feature learning while maintaining a ceiling on how much a noisy label can distort the weights.
Following a similar intuition, Feng et al. proposed Taylor Cross Entropy (TCE). By approximating CCE via a Taylor series expansion and controlling the order of the expansion, TCE acts as a tunable intermediate objective. It mitigates optimization difficulties on sophisticated architectures while retaining noise resilience.
Other information-theoretic formulations, such as the Generalized Jensen-Shannon (GJS) divergence loss proposed by Englesson et al., combine this interpolation concept with consistency regularization on perturbed inputs to ensure stable training without sacrificing speed.
Reshaping the Optimization Landscape
Rather than merely blending existing losses, another branch of work alters the optimization landscape itself—either by relaxing strict mathematical constraints or by modifying the network's output distributions.
Relaxing Symmetry and Normalization
While symmetric losses provide clean theoretical guarantees, their rigid structure is the primary cause of flat gradients. Zhou et al. relaxed this condition by introducing asymmetric loss functions. They proposed a metric called the asymmetry ratio, allowing designers to craft objectives that grant the model enough flexibility to fit complex patterns while remaining robust to label corruptions.
Alternatively, Ma et al. proved that any loss function can be made noise-robust simply by normalizing it. To counteract the underfitting inherent to normalized losses, they developed the Active Passive Loss (APL) framework. APL combines an "active" loss term (which maximizes the learning signal for clean data) with a "passive" loss term (which bounds the impact of noise). To improve upon the slow convergence of MAE-based passive components, Ye et al. introduced Normalized Negative Loss Functions (NNLFs), which accelerate training by focusing the optimization on well-learned clean samples.
Output Sharpening and Regularization
Instead of altering the loss equation directly, some approaches modify how the network outputs its predictions. Zhou et al. utilized sparse regularization to force the model's output probabilities toward a sharp, one-hot distribution, preventing the network from settling into the ambiguous, underfitted states induced by robust losses.
Expanding on this idea, Wang et al. developed $\epsilon$-softmax, a plug-and-play layer that forces network outputs to approximate one-hot vectors. This simple architectural modification allows standard loss functions to become inherently noise-tolerant without suffering from the flat-gradient bottlenecks typical of traditional robust alternatives.
Sample-Wise and Dynamic Adaptation
A final paradigm bypasses uniform loss application entirely, opting to dynamically adjust the optimization strategy based on the model's confidence in each individual sample.
In the framework developed by the authors of the Complementary Loss approach, a dual-loss strategy is deployed. Standard CCE is applied exclusively to high-confidence pseudo-labels to ensure rapid feature extraction, while a robust loss is applied to the original noisy labels to safely capture edge-case information without risk of memorization.
┌───────────────────────────┐
│ Incoming Sample │
└─────────────┬─────────────┘
│
[Evaluate Label Confidence]
│
┌───────────────┴───────────────┐
▼ ▼
High Confidence Low Confidence
[Apply Fast Loss: CE] [Apply Robust Loss: MAE]
│ │
└───────────────┬───────────────┘
▼
[Balanced Model Update]
Similarly, Ahn et al. incorporated an explicit sample-wise label confidence measure into the training loop. By selectively applying high-performance, sensitive losses to verified clean data and suppressing the influence of suspected mislabeled samples, they effectively bypassed the chronic underfitting problem. This strategy of combining internal model knowledge for data purification with robust objectives has expanded beyond traditional classification, proving vital for adapting large-scale vision-language models to noisy environments (Pan et al.).
Key Foundational Work
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018) — contributed 3 concept elements to this direction.
Robust Loss Functions under Label Noise for Deep Neural Networks (2017) — contributed 2 concept elements to this direction.
Taming the Cross Entropy Loss (2018) — contributed 1 concept elements to this direction.