Deep neural networks possess an extraordinary capacity to memorize data, a trait that becomes a major liability when training sets contain noisy or corrupted labels. Standard empirical risk minimization using Categorical Cross-Entropy (CCE) heavily penalizes large errors, forcing models to distort their decision boundaries to fit mislabeled outliers. As demonstrated by Deep Patel et al. in Memorization in Deep Neural Networks: Does the Loss Function matter?, models trained with standard objectives inevitably transition from learning generalizable features to memorizing random labels.
To combat this, a powerful mathematical framework has emerged around symmetric loss functions. By enforcing a simple structural constraint on the objective function, researchers can design algorithms that are inherently immune to label noise, allowing models to discover the true underlying data distribution without complex noise-estimation architectures.
The Intuition: Noise as a Constant Shift
To understand why symmetric loss functions work, we must look at how label noise alters the optimization landscape. A loss function $L(y, \hat{y})$ is defined as symmetric if the sum of the losses across all possible class labels for any given input $x$ is a constant $C$:
$$\sum_{k=1}^K L(k, \hat{y}) = C$$
When training data is corrupted by random or class-conditional label noise, the objective function we actually optimize (the noisy risk) becomes a mixture of the true clean risk and an error term stemming from the flipped labels.
If the loss function is symmetric, a remarkable algebraic cancellation occurs. As proven by Nontawat Charoenphakdee et al. in On Symmetric Losses for Learning from Corrupted Labels, the components of the risk associated with label corruption sum up to a constant value during optimization. Because the noise only introduces a constant shift to the objective function, it has zero effect on the gradients.
Consequently, gradient descent naturally ignores the noise and converges to the exact same global minimum it would have found using perfectly clean data.
Risk Ranking Invariance
This cancellation effect guarantees a property known as risk ranking invariance. As explored by Deep Patel et al., a symmetric loss function ensures that the relative performance ranking of different classifiers remains identical when moving from a clean evaluation set to a noisy training set. If Classifier A outperforms Classifier B on clean data, it will still outperform Classifier B on noisy data.
A classic example of an inherently symmetric objective is Mean Absolute Error (MAE). In Robust Loss Functions under Label Noise for Deep Neural Networks, Aritra Ghosh et al. proved that swapping CCE for MAE allows deep neural networks to maintain high predictive accuracy even in extreme scenarios where up to 80% of the training labels are incorrect.
Overcoming the Underfitting Bottleneck
While mathematically elegant, pure symmetric losses like MAE face significant practical hurdles in deep learning. Because their gradients do not scale with the magnitude of the prediction error, they often suffer from slow convergence and severe underfitting on complex datasets. To bridge the gap between theoretical robustness and practical optimization, several modern strategies have been developed:
- Symmetric Cross Entropy (SCE): Xingjun Ma et al. in Symmetric Cross Entropy for Robust Learning With Noisy Labels proposed combining standard CCE (which excels at training speed and fitting hard classes) with a noise-robust "Reverse Cross Entropy" (RCE) term. This hybrid approach maintains the optimization benefits of CCE while preventing the model from memorizing incorrect labels, resulting in tighter feature clusters in the latent space.
- Loss Normalization: Any standard loss function can be converted into a noise-robust variant through normalization. By dividing the error of a specific class by the sum of losses across all potential classes, the authors of Memorization in Deep Neural Networks: Does the Loss Function matter? demonstrate that models can be shielded from entering a noise-memorization phase. Similarly, Xichen Ye et al. in Active Negative Loss Functions for Learning with Noisy Labels utilize Normalized Negative Loss Functions to enforce symmetry while focusing the optimization signal on clean, well-learned samples.
- Output Approximation: Instead of modifying the loss directly, Jialiang Wang et al. introduced $\epsilon$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise. By forcing network outputs to approximate one-hot vectors, standard loss functions behave symmetrically during training, bypassing the typical optimization bottlenecks of robust objectives.
Applications Across Learning Paradigms
The utility of symmetric objectives extends far beyond standard image classification:
- Meta-Learning and Sample Weighting: In bilevel optimization frameworks designed to weight training samples dynamically, Aritra Ghosh et al. showed in Do We Really Need Gold Samples for Sample Weighting Under Label Noise? that symmetric losses align the expected gradient direction of a noisy dataset with that of a clean dataset. This allows meta-learning algorithms to function reliably without requiring expensive, perfectly curated "gold" validation sets.
- Adversarial Training: When defending networks against malicious attacks using datasets that contain inherent label noise, Zhe Chen et al. in Nrat: towards adversarial training with inherent label noise leverage symmetric loss formulations to bound the impact of mislabeled samples, preserving both adversarial robustness and standard generalization.
- Test-Time Adaptation: In unsupervised scenarios, Jun-Kun Wang et al. in Towards Understanding GD with Hard and Conjugate Pseudo-labels for Test-Time Adaptation demonstrate that conjugate pseudo-labels induce a symmetric objective function relative to the data distribution, helping gradient descent isolate the true class means from surrounding distribution noise.
Boundaries and Relaxations
While symmetric loss functions offer powerful guarantees for neural networks, they are not a universal remedy. Their behavior depends heavily on the underlying model architecture. For instance, Lukasz Sztukiewicz et al. in Exploring Loss Design Techniques For Decision Tree Robustness To Label Noise discovered that symmetric losses fail when applied to recursive partitioning models like decision trees. In that setting, the symmetry condition collapses the optimal leaf values into simple plurality indicators, causing zero-gain splits that prematurely halt tree growth.
Furthermore, strict mathematical symmetry can occasionally be overly restrictive for deep networks, leading to a drop in representation capacity. To address this, Xiong Zhou et al. in Asymmetric Loss Functions for Learning with Noisy Labels introduced a framework for asymmetric loss functions. By relaxing the strict symmetry constraint via a tunable "asymmetry ratio," the authors demonstrated that it is possible to retain noise tolerance while restoring convexity and strong fitting power, proving that strict symmetry is a sufficient—but not always necessary—condition for robust optimization.
Key Foundational Work
Learning with Noisy Labels (2013) — contributed 2 concept elements to this direction.
Learning with Noisy Labels (2013) — contributed 1 concept elements to this direction.
Analysis of Learning from Positive and Unlabeled Data (2014) — contributed 1 concept elements to this direction.