Robust Loss Functions for Learning with Imperfect Labels

51 concepts43 papers2 proposals

Deep neural networks possess a remarkable ability to memorize training data. While this allows them to learn complex patterns, it also creates a major vulnerability: if the training labels contain errors, the network will readily memorize these mistakes, severely degrading its generalization performance.

In ideal settings, we train models using standard loss functions like Cross-Entropy (CE) or Mean Squared Error (MSE). However, when dealing with real-world datasets—which are often plagued by human annotation errors, missing labels, or noisy pseudo-labels from distant supervision—standard objectives fail. To combat this, researchers have developed noise-robust loss functions. Rather than attempting to clean the data beforehand, these methods alter the optimization landscape itself, preventing the model from over-correcting when faced with incorrect supervision.


The Vulnerability of Standard Loss Functions

To understand why standard loss functions fail under noise, consider the behavior of Cross-Entropy loss for a classification task. The loss for a single sample is proportional to $-\log(p_y)$, where $p_y$ is the model's predicted probability for the given label $y$.

If a label is incorrect (noise), but the model has correctly identified the true underlying pattern, $p_y$ will be close to $0$. Because the derivative of $-\log(p_y)$ approaches infinity as $p_y$ goes to $0$, this single mislabeled sample generates an enormous gradient. This forces the model to drastically alter its weights to satisfy the erroneous label, leading to the memorization of noise.

A simple alternative is Mean Absolute Error (MAE). Unlike CE, the gradient magnitude of MAE with respect to the model's pre-softmax activation is bounded and constant, regardless of how far off the prediction is. As a result, an outlier or an incorrect label cannot dominate the gradient step.

Sebastian Bosse et al. demonstrated this advantage in Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment, utilizing MAE to achieve stable training despite the inherent variability and noise in human perceptual ratings. Similarly, Yanglin Feng et al. in RONO: Robust Discriminative Learning with Noisy Labels for 2D-3D Cross-Modal Retrieval showed that replacing CE with MAE provides a theoretically grounded resistance to label noise, keeping shared feature representations stable even when a significant portion of training labels are corrupted.


Balancing Convergence and Robustness

While MAE is inherently robust to noise, it suffers from a major practical drawback: it converges very slowly. Because its gradients do not scale down as the model gets closer to the correct prediction, optimization becomes difficult, particularly for complex architectures.

To capture the best of both worlds—the rapid convergence of CE and the noise-robustness of MAE—researchers use Generalized Cross-Entropy (GCE). GCE parameterizes the loss function using a Box-Cox transformation, allowing it to smoothly interpolate between CE and MAE via a tunable hyperparameter.

Standard Cross-Entropy (Fast convergence, high noise sensitivity)
      │
      ▼
Generalized Cross-Entropy (Tunable balance)
      ▲
      │
Mean Absolute Error (Slow convergence, high noise robustness)

This hybrid approach has proven highly effective across various domains:


Gradient Clipping via Partial Huberisation

Another elegant way to achieve robustness without abandoning Cross-Entropy entirely is to modify the loss function only when errors become excessively large. This is the core intuition behind partially Huberised cross-entropy.

For samples where the model's prediction aligns reasonably well with the label, the function behaves exactly like standard Cross-Entropy. However, when the error surpasses a certain threshold—indicating that the sample is likely a mislabeled outlier—the loss function switches from a logarithmic curve to a linear one.

$$\text{Loss} = \begin{cases} -\log(p_y) & \text{if } p_y \ge \tau \\ \text{Linear Extension} & \text{if } p_y < \tau \end{cases}$$

This linearization effectively clips the gradients for high-error samples, ensuring that unfilterable noise cannot force the model to overfit.

In self-training pipelines, where a model trains on its own pseudo-labels, this mechanism is vital. Austin Cheng-Yun Tsai et al. demonstrated in Contrast-Enhanced Semi-supervised Text Classification with Few Labels that linearizing the loss for high-error samples prevents the model from memorizing incorrect pseudo-labels. This strategy is mirrored by Chengyu Wang et al. in Uncertainty-Aware Self-Training for Low-Resource Neural Sequence Labeling to clip the influence of outliers that bypass initial confidence filters. Similarly, Paul Voigtlaender et al. utilized partially Huberised cross-entropy in Reducing the Annotation Effort for Video Object Segmentation Datasets to stabilize learning from imperfect, automatically generated segmentation masks.


Tailoring Losses to Specific Noise Patterns

Noise in machine learning is rarely uniform. Depending on the task, errors can manifest as missing labels (false negatives), multi-annotator ambiguity, or cross-modal mismatches. Researchers frequently design specialized loss functions to target these distinct structural patterns.

Asymmetric Penalties for Missing Labels

In automatically aligned or cross-lingual datasets, a common issue is the presence of false negatives—where a valid entity is simply not labeled. Standard loss functions penalize the model heavily for predicting a positive class when the label is negative. To fix this, Boyuan Zheng et al. introduced a noise-tolerant objective in Multilingual Coreference Resolution in Multiparty Dialogue that explicitly downweights the penalty for negative labels, preventing the model from being penalized when the alignment process fails to capture a valid mention.

Suppressing Large Discrepancies

In dense prediction tasks like crowd counting, human annotators frequently miss objects. Hui LIN et al. addressed this in Semi-supervised Crowd Counting via Density Agency by introducing a modulating factor into the loss function. This factor actively suppresses the influence of samples where the discrepancy between the prediction and the label is excessively large, preventing the model from overfitting to human omission errors.

Handling Multi-Annotator Disagreement

When multiple human annotators provide different labels for a single sample, treating any single label as the absolute ground truth introduces artificial noise. To resolve this ambiguity, Debaditya Roy in Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videos (and similarly explored in ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition) utilized a Minimum Annotator Cross Entropy Loss. Instead of penalizing the model for deviating from a single arbitrary choice, this loss calculates the cross-entropy against all valid human annotations and only penalizes the model relative to the closest one, preventing over-penalization during training.

Mismatched Modalities

In multimodal tasks, such as video retrieval, audio and visual channels are occasionally decoupled (e.g., a video of a person talking over unrelated background music). Sangmin Lee et al. addressed this in Audio-Visual Mismatch-Aware Video Retrieval via Association and Adjustment by designing an associative loss function with a logarithmic penalty that adaptively reduces the impact of pairs that do not naturally align, protecting the joint embedding space from corrupted multi-modal supervision.

Key Foundational Work

Making risk minimization tolerant to label noise (2015) — contributed 1 concept elements to this direction.
Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_noise_robust_loss_functions_571526dc,
  title = {noise-robust loss functions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/noise-robust-loss-functions/rdr_571526dcce2448c9a9cf6c256e71ac66},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_noise_robust_loss_functions_571526dc_v1,
  title = {noise-robust loss functions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/noise-robust-loss-functions/rdr_571526dcce2448c9a9cf6c256e71ac66/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake