Partial VOROS: A Cost-aware Performance Metric for Binary Classifiers with Precision and Capacity Constraints

Christopher Ratigan, Kyle Heuton, Carissa Wang, Lenore Cowen, Michael C. Hughes

TL;DR

Standard evaluation metrics like AUROC fail when real-world deployments face strict operational constraints, such as a minimum precision requirement to avoid alarm fatigue or maximum capacity limits on staff. This paper introduces Partial VOROS (PV), a novel, cost-aware performance metric that evaluates binary classifiers strictly within a mathematically defined "feasible region" of ROC space. By integrating over a distribution of asymmetric misclassification costs, PV helps practitioners select models that satisfy operational constraints while minimizing real-world costs.

What Problem Does This Paper Address?

When deploying machine learning models in high-stakes environments like hospitals (e.g., patient deterioration alert systems), standard metrics like the Area Under the ROC Curve (AUROC) fall short. They evaluate a model's performance across the entire ROC space, assuming all operating thresholds are equally viable.

In reality, clinicians face two major operational constraints:

  1. Precision Bounds: If an alert system has low precision (a high rate of false alarms), doctors and nurses suffer from "alarm fatigue" and eventually ignore the system entirely.
  2. Capacity Bounds: Hospital staff are a finite resource; they can only respond to a maximum number of total alerts per day.

Furthermore, standard metrics fail to account for asymmetric costs—missing a deteriorating patient (a false negative) is vastly more damaging than safely checking on a healthy patient due to a false alarm (a false positive). This paper addresses the challenge of effectively ranking and selecting binary classifiers when costs, performance constraints, and capacity limits must all be balanced simultaneously.

Methods

The authors build upon VOROS (Volume Over the ROC Surface), a recent metric that lifts the classic 2D ROC curve into a 3D space where the third axis represents varying misclassification costs ($t$).

To incorporate real-world constraints, the authors introduce the concept of a feasible region within ROC space. This region is bounded by two lines:

  • A minimum precision line ($\ell_\alpha$)
  • A maximum capacity line ($\ell_\kappa$)

They mathematically demonstrate that this feasible region forms a distinct polygon, falling into one of three geometric cases depending on how tightly the capacity constraint restricts the space.

Figure 1: Overview of VOROS (left) and our new partial VOROS (right). Left: At each fractional cost parameter t ∈[0, 1], we draw iso-performance lines for points P and Q in ROC space. Solid lines have lower cost than dashed lines. Points below a line have higher cost than that line. A point’s area of lesser classifiers, colored light pink here for point P, is the area below that point’s iso-performance line. The VOROS (Ratigan and Cowen, 2025) for a classifier is computed by finding at each t the maximum area of lesser classifiers for any point in its ROC curve, then integrating over a desired distribution p(t). Right: Partial VOROS excludes regions in gray that do not achieve a minimum precision α or exceed a maximum capacity κ (too many positive predictions). These limits correspond to linear constraints in ROC space.

The paper defines the Partial Area of Lesser Classifiers, which measures the area within this feasible polygon where alternative models would perform worse than the chosen classifier at a specific cost parameter $t$. By integrating this area over a probability distribution of expected costs, they compute the Partial VOROS (PV) metric. Crucially, the authors provide exact geometric formulas using the shoelace formula, allowing PV to be computed efficiently in $O(h_c)$ time, where $h_c$ is the number of vertices on the ROC curve's convex hull.

Datasets

The method was evaluated using real-world healthcare datasets for in-hospital mortality and deterioration prediction:

  • MIMIC-IV: A popular, open-access electronic health record (EHR) dataset. The authors extracted a cohort of over 31,000 ICU patient-stays to predict whether a patient would die during their hospital stay based on the first 48 hours of vital signs and lab measurements.
  • eICU Collaborative Research Database: A multi-center ICU database used as a secondary benchmark to evaluate model selection and generalization performance across more than 77,000 patient stays.
  • Edelson et al. (2024) Cohort: A large-scale dataset containing 362,926 patient stays across 7 hospitals, utilized for a post-hoc re-analysis of widely used clinical early warning scores.

Key Insights and Ideas

  • ROC Space Isn't All Usable: Enforcing precision and capacity constraints slices away large portions of the standard $[0,1] \times [0,1]$ ROC square. Evaluating models in unfeasible zones is a waste of optimization effort.
  • The Danger of Cost-Unaware Selection: Maximizing standard metrics like partial AUROC or recall within the feasible region can lead to suboptimal deployments. These metrics fail to account for how the optimal operating threshold shifts as the ratio of false-positive to false-negative costs changes.
  • Nuance Over Blanket Statements: Re-analyzing historical clinical scores showed that no single model dominates everywhere. A score that looks superior under loose constraints might be completely unusable under strict capacity limits.

Key Results

  • Nuanced Clinical Score Re-evaluation: In a re-analysis of established clinical scores (NEWS, NEWS2, and the Rothman Index), Partial VOROS revealed that the "best" model depends entirely on the specific precision and capacity thresholds chosen by a hospital, adding vital nuance to prior studies that declared a single blanket winner.
  • Lower Realized Costs: For model selection on MIMIC-IV and eICU, choosing models that maximized Partial VOROS consistently resulted in lower expected costs on the test set compared to cost-unaware alternatives (like maximizing recall or partial AUROC).
  • Constraint Enforcement: In strict deployment scenarios, ignoring constraints entirely (using original VOROS) selected models with significantly worse cost performance on test data.

Limitations and Future Work

  • Binary Classification Only: The metric is currently strictly limited to binary classification tasks; extending it to multi-class or multi-label settings remains future work.
  • Prevalence Sensitivity: Because precision is inherently tied to class balance, PV values cannot be compared across datasets with different base rates (prevalence) of the positive class.
  • Elicitation Challenge: It requires stakeholders to explicitly define hard thresholds ($\alpha, \kappa$) and a cost distribution, which can be difficult for non-technical domain experts to estimate accurately.
  • Statistical Generalization: Due to statistical noise, a model meeting constraints perfectly on validation data can occasionally violate them slightly when deployed on new test data. The authors suggest that future work should explore statistical generalization bounds or slightly over-enforce constraints during validation to guarantee safety.

Summary updated

Abstract

The ROC curve is widely used to assess binary classifiers. Yet for some applications, such as alert systems for monitoring hospitalized patients, conventional ROC analysis cannot meet two key deployment needs: enforcing a constraint on precision to avoid false alarm fatigue and imposing an upper bound on the number of predicted positives to represent the capacity of hospital staff. The usual area under the curve metric also does not reflect asymmetric costs for false positives and false negatives. In this paper we address all three of these issues. First, we show how the subset of classifiers that meet precision and capacity constraints occupy a feasible region in ROC space. We establish the polygon-shaped geometry of this region. We then define the partial area of lesser classifiers, a performance metric that is monotonic with cost and only accounts for the feasible region. Averaging this area over a desired distribution for cost parameters results in the partial volume over the ROC surface, or partial VOROS. In experiments predicting mortality risk from vital sign history on several datasets, we show this cost-aware metric can outperform alternatives at ranking classifiers for in-hospital alerts.
BibTeX
@inproceedings{ratigan2026partial,
  author = {Christopher Ratigan and Kyle Heuton and Carissa Wang and Lenore Cowen and Michael C. Hughes},
  title = {{Partial VOROS: A Cost-aware Performance Metric for Binary Classifiers with Precision and Capacity Constraints}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=9J9Ly2xUxL},
}
Figure