Designing off-sample performance metrics

Matthew J. HollandarXiv 2021 · arXiv · OpenReview

Abstract

Modern machine learning systems are traditionally designed and tested with the overall goal of achieving the best possible performance on average. In this work, we consider an approach to building learning systems which treats the question of "how should we quantify good off-sample performance?" as a key design decision. We describe this proposal using a simple and general formulation, place the current dominant paradigm within the proper historical context, and then survey the literature for more recent developments that depart from tradition and can be viewed as special cases of our proposed methodology.

Introduction

In the classical paradigm of statistical machine learning, "good performance" is almost universally synonymous with minimizing the expected loss (the sample mean) on a test set. While this approach is mathematically convenient and has driven significant progress, it suffers from critical blind spots. An obsession with average-case performance often ignores heavy-tailed distributions, where extreme outliers can skew results; representation disparity, where a model performs well on average but fails specific sub-groups; and sensitivity to distribution shift, where minor changes in data environment lead to catastrophic failure.

This paper argues that the choice of a performance metric should not be a tacit default, but rather a deliberate design decision. By moving beyond the mean, practitioners can better align machine learning objectives with real-world requirements such as fairness, risk aversion, and robustness.

The Generalized Learning Problem

To address the limitations of the average-case paradigm, we introduce a Generalized Learning Problem framework. In standard learning, we seek a decision $h$ that minimizes the risk $E[loss]$. We generalize this objective to a requirement that a specific property of the loss distribution meets a desirability threshold:

$$\mathbf{P} \{C_\mu(H_n) \in \mathcal{C}\} \geq 1 - \delta$$

In this formulation, $C_\mu$ is a learning criterion function—a property of the loss distribution such as its median, variance, or tail weight—and $\mathcal{C}$ represents a desirability set. This framework allows us to categorize the landscape of non-traditional performance metrics into three primary families:

  1. Order-Based Criteria: These metrics involve sorting losses and weighting them based on their rank. Examples include "Average Top-k" loss or trimmed means, which are particularly useful for ignoring outliers or focusing on the most difficult examples.
  2. Optimized Certainty Equivalent (OCE) Risks: Derived from economic theory, OCE risks create a formal trade-off between certain and uncertain outcomes. This family includes Conditional Value-at-Risk (CVaR) and "tilted" (entropic) risks, which prioritize the tail of the loss distribution to avoid catastrophic failures.
  3. Distributionally Robust Optimization (DRO): DRO criteria seek to minimize the worst-case expected loss over a set of distributions "near" the observed data. This provides a mathematical safety buffer against potential distribution shifts.

Bridging Theory and Practice

DRO as Variance Regularization

A significant insight from this framework is the mathematical link between Distributionally Robust Optimization and variance control. Minimizing a DRO objective is often equivalent to minimizing a combination of the mean and the standard deviation. This connection demonstrates that robustness is not just a theoretical ideal but a practical method for achieving more stable and reliable model performance.

Fairness as a Distributional Property

Rather than treating fairness as an external constraint or a post-processing step, our framework reframes fairness as a specific choice of learning criterion. By selecting a criterion that penalizes high variance in risk across different sub-groups, fairness becomes a natural byproduct of the optimization process. This integrates social objectives directly into the mathematical heart of the model.

Human-Aligned Risks

Drawing from behavioral economics, we explore Cumulative Prospect Theory (CPT) as a basis for performance metrics. CPT models how humans perceive risk—specifically, our tendency to overweight small probabilities of large losses. By adopting psychologically inspired criteria, machine learning models can be better aligned with human values and risk tolerances in sensitive fields like medicine and finance.

Discussion and Future Directions

While the shift toward intentional metric design offers significant benefits, several challenges remain. Many non-traditional criteria, such as quantiles or M-parameters, are computationally difficult to optimize because they can be non-convex or non-smooth. Furthermore, the choice of a criterion remains a subjective value judgment; there is no universal "best" metric, only the metric most appropriate for a specific application.

Additionally, the interaction between these non-linear criteria and the "surrogate" losses (like cross-entropy) commonly used in deep learning is not yet fully understood. Future work must focus on developing principled, transparent procedures for criteria design, ensuring that engineers have the tools to choose the right mathematical yardstick for their specific real-world goals. By treating off-sample performance as a design choice, we can build systems that are not just accurate on average, but robust and fair in practice.

Summary updated

Abstract

Modern machine learning systems are traditionally designed and tested with the overall goal of achieving the best possible performance on average. In this work, we consider an approach to building learning systems which treats the question of "how should we quantify good off-sample performance?" as a key design decision. We describe this proposal using a simple and general formulation, place the current dominant paradigm within the proper historical context, and then survey the literature for more recent developments that depart from tradition and can be viewed as special cases of our proposed methodology.
BibTeX
@misc{holland2021designing,
  author = {Matthew J. Holland},
  title = {{Designing off-sample performance metrics}},
  year = {2021},
  eprint = {2110.04996},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.04996},
}
Figure