In standard machine learning, we typically minimize the average loss across our dataset via Empirical Risk Minimization (ERM). When we do care about risk sensitivity—such as in financial forecasting, robotics, or safety-critical systems—we usually focus exclusively on the upper tail of the loss distribution. We want to prevent catastrophic, high-loss failures.
However, focusing solely on one side of the distribution creates a dangerous theoretical and practical blind spot. If an objective or a guarantee only constrains the upper tail, the lower tail is left entirely unmanaged. A model can become overly confident by exploiting "lucky" training samples that yield unusually small losses, leading to poor stability and brittle generalization. To build truly reliable systems, a growing body of research focuses on controlling the entire shape of the loss distribution by looking both ways: penalizing deviations in both the upper and lower tails, and proving two-sided guarantees.
Shaping the Loss Distribution via Bidirectional Penalization
How do we design optimization objectives that explicitly manage both tails? Traditional risk measures lack the flexibility to control the holistic shape of the loss distribution. To address this, Matthew J. Holland introduced a framework based on M-location estimators in Learning with risks based on M-location. Instead of just minimizing high losses, this approach employs a bidirectional penalization strategy that accounts for deviations in both directions.
By discouraging both excessively large and unusually small losses, the risk function encourages the resulting loss distribution to be symmetric and tightly concentrated around a central value. This prevents the model from ignoring unusually small losses—a critical requirement when dealing with loss functions that are unbounded below.
Building on this concept, in Flexible risk design using bi-directional dispersion, Matthew J. Holland proposed "T-risks" (Threshold risks). T-risks penalize loss deviations both above and below a specific target threshold.
Why does this work so well in practice? By penalizing the lower tail, the model is prevented from overfitting to lucky outliers or developing extreme overconfidence. Simultaneously, managing the upper tail mitigates the impact of high-loss outliers. The result is a highly stable classification performance, even when training on noisy or heavy-tailed real-world data.
The Necessity of Two-Sided Generalization Bounds
The value of bidirectional control extends far beyond designing better loss functions; it is equally vital for proving theoretical guarantees. In machine learning theory, we rely on generalization and concentration bounds to tell us how close our empirical risk estimate is to the true, unseen risk.
Historically, many risk-sensitive frameworks relied on one-sided generalization bounds. These bounds only guarantee that the true risk will not exceed a certain upper limit. While this sounds safe, it introduces a severe estimator reliability gap. If a bound lacks a lower limit, the empirical estimator can drastically under-report the risk (or over-report rewards) without violating the theoretical guarantee.
As demonstrated by Imad AOUALI et al. in Exponential Smoothing for Off-Policy Learning, one-sided bounds are fundamentally insufficient for policy optimization. In off-policy learning, estimators like Inverse Propensity Scoring (IPS) suffer from high variance. If you optimize an estimator using only a one-sided upper bound, the optimization process can easily converge on trivial or inaccurate estimators that satisfy the upper bound but provide no real predictive power. By deriving a novel, tractable two-sided PAC-Bayes generalization bound, the authors ensure that the performance estimate is tightly squeezed from both above and below. This guarantees that the empirical estimator is approximately correct, making it safe to use directly as an optimization objective.
Mathematical Foundations of Two-Sided Concentration
To trust these two-sided guarantees across diverse and potentially unbounded data distributions (such as sub-Gaussian or sub-exponential variables), we need a robust mathematical framework to quantify how risk estimates concentrate.
In Concentration of risk measures: A Wasserstein distance approach, Sanjay P. Bhat et al. tackled this by leveraging the geometric properties of the Wasserstein distance. The authors derived tight, two-sided concentration bounds for several popular risk measures.
These bounds establish both upper and lower limits on the probability of deviation between the estimated and actual risk. In practical applications like multi-armed bandits or risk-sensitive reinforcement learning, this symmetry is essential. It allows practitioners to construct robust confidence intervals where overly conservative estimates (overestimating risk) are balanced against safety requirements (underestimating risk).
Summary
Ultimately, the shift toward bidirectional tail penalization and two-sided concentration bounds reflects a deeper realization in machine learning: true robustness requires holistic distribution shaping. Whether we are designing a loss function to prevent model overconfidence or proving a generalization bound to ensure an estimator is informative, we cannot afford to leave one tail unconstrained. Squeezing the distribution from both sides is what turns unstable, volatile estimators into reliable objectives for optimization.
Key Foundational Work
Concentration of risk measures: A Wasserstein distance approach (2019) — contributed 3 concept elements to this direction.
Learning with risks based on M-location (2022) — contributed 2 concept elements to this direction.