Symmetric Loss Functions for Robust Label Noise Learning

This direction is split into other directions: symmetric loss functions. Its last version stays readable and citable here.

Version 1 of 1108 papers174 conceptsfrozen 2026-10-04

Deep neural networks are notorious for their ability to memorize training data, even when that data is intentionally corrupted with random labels. In a perfect world, we would train on "gold standard" datasets. In reality, we often rely on crowdsourced annotations, automated pseudo-labels, or web-scraped data—all of which are inherently noisy. When a model encounters a mislabeled sample, a standard loss function like Cross-Entropy (CE) penalizes the "error" so aggressively that the model eventually warps its decision boundary to fit the noise, ruining its ability to generalize to clean test data.

To combat this, researchers have developed Symmetric Loss Functions. The core intuition is simple: if a loss function treats all classes with a certain mathematical "balance," the noise effectively cancels itself out during optimization.

The Symmetry Condition

A loss function $L$ is considered symmetric if the sum of the losses over all possible classes $K$ is constant for any prediction. Mathematically, for a classifier $f(x)$:

$$\sum_{j=1}^K L(f(x), j) = C$$

Why does this matter? Nagarajan Natarajan, Ambuj Tewari, and Pradeep Ravikumar demonstrated in Learning with Noisy Labels that when a loss function satisfies this symmetry, its expected value on noisy data becomes an affine transformation of its value on clean data.

This is a profound result. It means that the global minimizer—the "perfect" version of the model—remains the same whether you train on clean data or data corrupted by class-conditional noise. As Nontawat Charoenphakdee and Masashi Sugiyama argue in their work on one-class classification, this property prevents the model from overfitting to systematic errors introduced by imperfect labeling heuristics.

Balancing Attraction and Repulsion

The most common symmetric loss is Mean Absolute Error (MAE). However, MAE is notoriously difficult to train with deep networks because its gradients don't scale with the magnitude of the error, leading to agonizingly slow convergence.

To solve this, Xingjun Ma and Yisen Wang proposed Symmetric Cross Entropy (SCE) in Symmetric Cross Entropy for Robust Learning With Noisy Labels. SCE combines the standard Cross-Entropy (CE) with a "Reverse Cross Entropy" (RCE) term.

While CE pushes the model to maximize the probability of the (potentially wrong) label, RCE acts as a noise-robust anchor. In the "Attract-and-Repulse" framework described by Shicai Yang and Weijie Chen in An Attract-and-Repulse Learning Approach, this combination balances the learning process. The CE term provides the necessary "attraction" to learn complex patterns quickly, while the symmetric RCE term provides a "repulsive" force that prevents the model from over-focusing on outliers or ambiguous samples.

Peer Loss: Robustness Without Noise Rates

Most robust methods require knowing the "noise rate"—exactly how often the labels are wrong. In practice, this is almost impossible to estimate accurately.

Yang Liu and Hongyi Guo introduced Peer Loss in Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates to bypass this requirement. Peer Loss works by taking a standard loss and subtracting the loss calculated against a "peer" sample—a random label from the dataset.

The intuition is purely mathematical: by subtracting the performance on a random label, you cancel out the bias introduced by the noise in the joint distribution of features and labels. This makes the optimization invariant to the noise rate, effectively allowing the model to "see through" the corruption to the underlying clean signal.

Information-Theoretic Invariance

Another perspective on symmetry comes from information theory. Yilun Xu and Yizhou Wang proposed $L_{DMI}$ in An Information-theoretic Noise-robust Loss Function. This loss uses the determinant of the joint probability matrix between predicted and observed labels.

Because the determinant of a product of matrices is the product of their determinants, the noise transition matrix becomes a constant multiplier that doesn't change the location of the optimal parameters. This provides a theoretical guarantee: the model will converge to the same state whether the labels are 100% clean or 40% noisy, provided the noise is instance-independent.

Relaxing the Symmetry Constraint

While strict symmetry provides beautiful theoretical guarantees, it can sometimes lead to underfitting because it limits the model's "fitting power." Recent work has sought to relax these constraints.

Xiong Zhou and Xianming Liu introduced Asymmetric Loss Functions in Asymmetric Loss Functions for Learning with Noisy Labels, arguing that symmetry is a sufficient but not strictly necessary condition for robustness. By using an "asymmetry ratio," they design losses that are easier to optimize while still maintaining "classification calibration"—the property that the noisy minimizer matches the clean one.

Similarly, Jialiang Wang and Xiong Zhou proposed Variation-Bounded Loss in Variation-Bounded Loss for Noise-Tolerant Learning. Instead of strict symmetry, they bound the ratio of the maximum and minimum gradients. This prevents the model from over-prioritizing the high-gradient "hard" samples that are often just mislabeled noise, without sacrificing the ability to learn from legitimate hard cases.

Beyond Classification: Contrastive Learning

The principles of symmetry are now moving into self-supervised and contrastive learning. In Robust Contrastive Learning against Noisy Views, Ching-Yao Chuang and Antonio Torralba show that symmetric losses naturally downweight "noisy views"—pairs that are supposed to be similar but aren't.

Building on this, Jingyi Cui and Yisen Wang developed Symmetric InfoNCE (SymNCE) in An Inclusive Theoretical Framework of Robust Supervised Contrastive Loss. By symmetrizing the contrastive objective, they ensure that the risk introduced by label corruption remains constant relative to the model's parameters, preventing incorrect labels from degrading the quality of learned feature representations.

Papers in this version

Showing 60 of 108 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_symmetric_loss_functions_for_robust_labe_88a6deb0,
  title = {Symmetric Loss Functions for Robust Label Noise Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symmetric-loss-functions-for-robust-label-noise-learning/rdr_88a6deb027574d41afedec78cfa89449},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_symmetric_loss_functions_for_robust_labe_88a6deb0_v1,
  title = {Symmetric Loss Functions for Robust Label Noise Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symmetric-loss-functions-for-robust-label-noise-learning/rdr_88a6deb027574d41afedec78cfa89449/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}