Introduction
In the standard machine learning paradigm, success is almost exclusively measured by the minimization of average loss across a dataset—a framework known as Expected Risk Minimization (ERM). While ERM is mathematically elegant and computationally efficient, it is often treated as a default rather than a deliberate choice. This survey challenges this status quo, arguing that by focusing solely on the mean, researchers ignore the broader distribution of losses, potentially overlooking critical issues related to safety, fairness, and robustness.
The Limitations of Expected Risk
The "average" is a convenient summary statistic, but it is often a poor proxy for reliability in high-stakes scenarios. A model with a low average loss might still exhibit catastrophic failures on specific subsets of data or be highly sensitive to outliers. In fields like healthcare, autonomous driving, or finance, the "tails" of the loss distribution—where the worst-case outcomes reside—are often more important than the mean performance.
By relying on ERM, practitioners make a tacit trade-off, prioritizing general performance over tail-risk management. This paper seeks to formalize the alternatives, providing a roadmap for criteria that allow for more nuanced definitions of "desirable performance."
From Minimizing Numbers to Shaping Distributions
The core contribution of this work is a shift in perspective: loss should not be viewed as a single scalar value to be minimized, but as a random variable with its own distribution. Under this view, the goal of a learning algorithm is to "shape" this distribution.
The authors categorize various non-traditional learning criteria into a structured taxonomy. These include:
- Distributionally Robust Optimization (DRO): Designing models that perform well under the worst-case distribution within a certain neighborhood of the training data.
- Risk-Sensitive Objectives: Utilizing metrics like Conditional Value-at-Risk (CVaR) to focus specifically on the upper quantiles of the loss distribution.
- Variance-Regularized Objectives: Balancing the mean loss with its variance to ensure more consistent performance across the dataset.
Systematizing the Field
As a survey and abstract reprint, the paper does not aim to provide new quantitative benchmarks. Instead, its value lies in the qualitative systematization of the field. It contextualizes the classical paradigm as just one point in a vast landscape of possible learning objectives. By providing a conceptual framework, the authors enable researchers to move beyond the "average trap" and select criteria that align with the specific requirements of their application—whether that be robustness to distribution shift or ensuring equitable performance across demographic groups.
Challenges in Implementation
Moving beyond the usual risk is not without its hurdles. The authors identify two primary limitations that currently hinder the widespread adoption of these criteria:
- Theoretical Complexity: Many non-traditional criteria lack the "convenient" mathematical properties (such as simple convergence proofs) that make ERM easy to analyze.
- Optimization Difficulty: These criteria often result in more complex optimization landscapes compared to standard gradient descent on average loss, requiring more sophisticated algorithmic approaches.
Conclusion
The choice of a learning criterion is a fundamental design decision that dictates how a model perceives "goodness." This survey serves as a call to action for the machine learning community to be more intentional in this choice. By viewing learning as a task of distribution shaping rather than simple mean minimization, we can build models that are not just accurate on average, but robust and reliable in the real world.
Summary updated