A Survey of Learning Criteria Going beyond the Usual Risk (Abstract Reprint)

Matthew J. Holland, Kazuki Tanabe

Introduction

In the standard machine learning paradigm, success is almost exclusively measured by the minimization of average loss across a dataset—a framework known as Expected Risk Minimization (ERM). While ERM is mathematically elegant and computationally efficient, it is often treated as a default rather than a deliberate choice. This survey challenges this status quo, arguing that by focusing solely on the mean, researchers ignore the broader distribution of losses, potentially overlooking critical issues related to safety, fairness, and robustness.

The Limitations of Expected Risk

The "average" is a convenient summary statistic, but it is often a poor proxy for reliability in high-stakes scenarios. A model with a low average loss might still exhibit catastrophic failures on specific subsets of data or be highly sensitive to outliers. In fields like healthcare, autonomous driving, or finance, the "tails" of the loss distribution—where the worst-case outcomes reside—are often more important than the mean performance.

By relying on ERM, practitioners make a tacit trade-off, prioritizing general performance over tail-risk management. This paper seeks to formalize the alternatives, providing a roadmap for criteria that allow for more nuanced definitions of "desirable performance."

From Minimizing Numbers to Shaping Distributions

The core contribution of this work is a shift in perspective: loss should not be viewed as a single scalar value to be minimized, but as a random variable with its own distribution. Under this view, the goal of a learning algorithm is to "shape" this distribution.

The authors categorize various non-traditional learning criteria into a structured taxonomy. These include:

  • Distributionally Robust Optimization (DRO): Designing models that perform well under the worst-case distribution within a certain neighborhood of the training data.
  • Risk-Sensitive Objectives: Utilizing metrics like Conditional Value-at-Risk (CVaR) to focus specifically on the upper quantiles of the loss distribution.
  • Variance-Regularized Objectives: Balancing the mean loss with its variance to ensure more consistent performance across the dataset.

Systematizing the Field

As a survey and abstract reprint, the paper does not aim to provide new quantitative benchmarks. Instead, its value lies in the qualitative systematization of the field. It contextualizes the classical paradigm as just one point in a vast landscape of possible learning objectives. By providing a conceptual framework, the authors enable researchers to move beyond the "average trap" and select criteria that align with the specific requirements of their application—whether that be robustness to distribution shift or ensuring equitable performance across demographic groups.

Challenges in Implementation

Moving beyond the usual risk is not without its hurdles. The authors identify two primary limitations that currently hinder the widespread adoption of these criteria:

  1. Theoretical Complexity: Many non-traditional criteria lack the "convenient" mathematical properties (such as simple convergence proofs) that make ERM easy to analyze.
  2. Optimization Difficulty: These criteria often result in more complex optimization landscapes compared to standard gradient descent on average loss, requiring more sophisticated algorithmic approaches.

Conclusion

The choice of a learning criterion is a fundamental design decision that dictates how a model perceives "goodness." This survey serves as a call to action for the machine learning community to be more intentional in this choice. By viewing learning as a task of distribution shaping rather than simple mean minimization, we can build models that are not just accurate on average, but robust and reliable in the real world.

Summary updated

Abstract

Virtually all machine learning tasks are characterized using some form of loss function, and "good performance" is typically stated in terms of a sufficiently small average loss, taken over the random draw of test data. While optimizing for performance on average is intuitive, convenient to analyze in theory, and easy to implement in practice, such a choice brings about trade-offs. In this work, we survey and introduce a wide variety of non-traditional criteria used to design and evaluate machine learning algorithms, place the classical paradigm within the proper historical context, and propose a view of learning problems which emphasizes the question of "what makes for a desirable loss distribution?" in place of tacit use of the expected loss.
BibTeX
@inproceedings{holland2024survey,
  author = {Matthew J. Holland and Kazuki Tanabe},
  title = {{A Survey of Learning Criteria Going beyond the Usual Risk (Abstract Reprint)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30598},
}
Figure