Abstract
In this work, we consider the setting of learning problems under a wide class of spectral risk (or "L-risk") functions, where a Lipschitz-continuous spectral density is used to flexibly assign weight to extreme loss values. We obtain excess risk guarantees for a derivative-free learning procedure under unbounded heavy-tailed loss distributions, and propose a computationally efficient implementation which empirically outperforms traditional risk minimizers in terms of balancing spectral risk and misclassification error.
Introduction
Standard machine learning approaches typically rely on Empirical Risk Minimization (ERM), which optimizes for the average performance of a model. However, in high-stakes domains such as finance or safety-critical engineering, the "average" case is often insufficient. Practitioners are frequently more concerned with the "tail" of the loss distribution—the rare but catastrophic failures.
While Conditional Value-at-Risk (CVaR) is a common tool for risk-aware learning, it lacks flexibility by treating all losses above a fixed threshold equally. Spectral risks offer a more sophisticated alternative, allowing for a custom weighting function across the entire loss distribution. Despite their theoretical appeal, spectral risks are challenging to optimize due to their non-smooth nature and sensitivity to heavy-tailed distributions. This paper introduces a robust framework to optimize these risks efficiently, even when losses are unbounded.
Methodology
The authors propose a multi-stage approach to handle the complexities of spectral risk optimization:
- Derivative-Free Optimization with Stochastic Smoothing: To address the non-smoothness of spectral risks, the researchers employ a "stochastic smoothing" technique. By injecting small amounts of random noise into model parameters, they create a differentiable surrogate risk. A 0th-order gradient estimator is then used within a Mirror Descent framework.
- Robust Confidence Boosting: To mitigate the high variance inherent in heavy-tailed data, the algorithm generates multiple candidate models. A robust M-estimator is then used to select the final model, effectively filtering out the influence of extreme outliers.
- The "Fast" Implementation: Recognizing the computational overhead of derivative-free methods, the authors developed a modified procedure. This version uses a folded Normal distribution to approximate the loss distribution, enabling the integration of actual gradient information and significantly accelerating convergence.
Empirical Evaluation
The framework was tested across eight diverse benchmark datasets, ranging from tabular data (adult, protein) to image classification (cifar10, fashion_mnist). The evaluation focused on two primary metrics: the spectral risk itself and the resulting misclassification error.
A key finding was the surprising effectiveness of the "Fast" implementation. Not only did it manage risk effectively, but it also frequently achieved lower misclassification errors than standard ERM. This suggests that incorporating risk-awareness can act as a powerful regularizer, improving general model performance.
Key Insights and Conclusion
This research demonstrates that spectral risks can be optimized effectively without assuming bounded losses. The theoretical derivation of a sample complexity of $O(\epsilon^{-8})$ provides a solid foundation for derivative-free learning in this space. Empirically, the "Fast" parametric approach offers a practical, high-performance alternative to traditional ERM, showing that robustness to extreme losses does not have to come at the cost of average-case accuracy.
Future work will look toward improving the convergence rates of derivative-free estimators and extending these risk-aware guarantees to the complex loss landscapes of deep neural networks.
Summary updated