Bounded Loss Functions for Outlier Robustness

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2109 papers122 conceptsfrozen 2026-10-04

In machine learning, convexity is often treated as a non-negotiable requirement for loss functions. If a problem is convex, we can guarantee finding a global optimum efficiently. However, this mathematical convenience comes with a hidden cost: most standard convex losses—like Cross-Entropy, Hinge loss, and Mean Squared Error—are unbounded.

As a model’s prediction moves further from the true label, an unbounded loss function applies a penalty that grows indefinitely. While this helps the model learn from clear signals, it also makes the model hypersensitive to outliers and noisy labels. A single extreme data point can generate a massive gradient that "hijacks" the optimization process, forcing the decision boundary to shift significantly just to accommodate a likely error.

A growing body of research suggests that by trading convexity for boundedness, we can build models that are inherently robust to "dirty" data.

The Sensitivity of Unbounded Losses

The fundamental problem with standard losses is their lack of a "ceiling." In Trading convexity for scalability, Ronan Collobert, Jason Weston, and Leon Bottou argue that the Hinge loss used in Support Vector Machines (SVMs) forces the model to treat extreme outliers as support vectors. This not only degrades accuracy but also increases model complexity (sparsity) unnecessarily.

This issue is even more pronounced in deep learning. Hongxin Wei and colleagues demonstrate in Mitigating Memorization of Noisy Labels by Clipping the Model Prediction that Cross-Entropy’s unbounded nature is what allows neural networks to "memorize" noisy labels. Because the loss approaches infinity as the predicted probability of a label approaches zero, the optimizer is forced to prioritize fitting these high-loss outliers at the expense of generalizable features.

Capping the Penalty: Ramp and Truncated Losses

The solution is to use loss functions that "saturate." Once a data point is far enough on the wrong side of the decision boundary, the model should stop trying to fix it.

  1. Ramp Loss: This is a "clipped" version of the Hinge loss. Once the error reaches a certain threshold, the penalty stays constant. Nam Ho-Nguyen shows in Adversarial classification via distributional robustness with Wasserstein ambiguity that minimizing Ramp loss is equivalent to achieving distributional robustness, protecting the model against perturbations in the data distribution.
  2. Loss Truncation: In natural language generation, standard log-loss forces models to mimic every example, including hallucinations or noise. Daniel Kang and Tatsunori Hashimoto introduced Loss Truncation, which adaptively drops high-loss examples during training. This simple change allows the model to ignore noisy data, leading to significantly higher factual accuracy in summarization tasks.
  3. $\alpha$-Loss: Tyler Sypherd and Gautam Dasarathy proposed $\alpha$-loss, a tunable framework that bridges the gap between log-loss and the ideal (but non-differentiable) 0-1 loss. By adjusting $\alpha$, practitioners can choose to ignore outliers ($\alpha > 1$) or focus on rare classes ($\alpha < 1$).

Benign Non-Convexity

The primary argument against bounded losses is that they are non-convex, which usually implies a landscape full of treacherous local minima. However, recent theoretical work suggests this fear may be overstated.

Nam Ho-Nguyen proves that for many common data distributions, the non-convexity of the Ramp loss is "benign"—meaning standard optimization algorithms can still reliably find the global best solution. Similarly, Nan Ding and S. V. N. Vishwanathan introduced t-logistic regression, which uses a non-convex loss to downweight outliers. They found that while the loss is non-convex, it is "quasi-convex" under certain conditions, making it computationally tractable while remaining far more robust than standard logistic regression.

Robustness Across Domains

The shift toward bounded losses is appearing across diverse subfields:

  • Bayesian Learning: Osvaldo Simeone and Marios Kountouris developed $(m, t)$-robust Bayesian learning, which uses "tempered" logarithms to prevent a single outlier from exerting infinite influence on the posterior distribution.
  • Fairness: Cuong Tran and Keyu Zhu found that standard unbounded losses exacerbate the trade-off between fairness and adversarial robustness. In Fairness Increases Adversarial Vulnerability, they propose using Ramp loss to prevent outliers from pulling decision boundaries into vulnerable, unfair positions.
  • Preference Optimization: When aligning LLMs with human feedback, Yu-Jie Zhang and Masashi Sugiyama argue in Symmetric Preference Optimization that standard convex losses like DPO are too sensitive to noisy human labels, and that symmetric, bounded losses are necessary for stable alignment.

By accepting the challenge of non-convex optimization, these researchers have shown that we can build models that focus on the "clean" majority of the data, effectively ignoring the noise that would otherwise derail a standard convex model.

Papers in this version

Showing 60 of 109 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 66 papers
  2. Version 1 · 2026-10-04 · 109 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_convex_loss_sensitivity_4e209b7a,
  title = {convex loss sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/convex-loss-sensitivity/rdr_4e209b7abe704e02ab00990f243c5e94},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_bounded_loss_functions_for_outlier_robus_4e209b7a_v1,
  title = {Bounded Loss Functions for Outlier Robustness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bounded-loss-functions-for-outlier-robustness/rdr_4e209b7abe704e02ab00990f243c5e94/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}