The VOROS: Lifting ROC Curves to 3D to Summarize Unbalanced Classifier Performance

Christopher Ratigan, Lenore CowenAAAI 2025 · DOI

Introduction: The Blind Spot of AUROC

In machine learning, the Area Under the Receiver Operating Characteristic curve (AUROC) is the standard metric for evaluating binary classifiers. However, AUROC has a significant limitation: it assumes that False Positives (FP) and False Negatives (FN) carry equal costs and that the dataset is balanced. In critical domains like medical diagnosis or fraud detection, these assumptions rarely hold. Missing a cancer diagnosis (FN) is typically far more costly than a false alarm (FP), and fraudulent transactions are often a tiny fraction of total data.

When costs and class distributions are imbalanced, a classifier with a higher AUROC may actually perform worse in a real-world deployment than one with a lower AUROC. While weighted AUROC methods exist, they require precise knowledge of cost and imbalance ratios beforehand. This paper introduces VOROS (Volume Over the ROC Surface), a metric that generalizes AUROC by lifting it into 3D to account for varying or uncertain misclassification costs and class imbalances.

Lifting ROC Curves to 3D

The core innovation of VOROS is the introduction of a third axis, $t$, representing the "normalized expected cost." This parameter $t$ (ranging from 0 to 1) encapsulates both the ratio of misclassification costs and the class imbalance into a single value. By adding this dimension, the standard 2D ROC space is transformed into a 3D volume.

The ROC curve lifted into a 3D space where the t-axis represents normalized expected cost.

The VOROS Metric: Volume Over the ROC Surface

For any given cost $t$, there is a specific region in the ROC space that represents performance worse than the optimal point on the ROC curve. VOROS is defined by integrating these "areas of lesser classifiers" across the entire $t$-axis, creating a 3D volume.

Mathematically, VOROS represents the probability that a randomly chosen deployment scenario (defined by cost and imbalance) and a random classifier point will result in a higher cost than the model's optimal threshold. The authors demonstrate that this volume can be calculated efficiently in $O(n \log n)$ time using the upper convex hull of the ROC points.

The relationship between the cost parameter t and the area A, which together define the VOROS volume.

One of the most powerful features of VOROS is range-based evaluation. If a domain expert knows that FNs are significantly more expensive than FPs, they can calculate the volume over a specific sub-interval of $t$, allowing for a cost-aware ranking of classifiers even when exact ratios are unknown.

Empirical Validation: Medical and Fraud Detection

The authors validated VOROS across several benchmark datasets, revealing insights that standard AUROC misses:

  • Medical Data (Wisconsin Breast Cancer): While Logistic Regression and Random Forest showed nearly identical AUROCs (99.3% vs 99.0%), VOROS revealed that Random Forest is superior in specific low-cost regimes ($t \in [0, 0.25]$).
  • Fraud Detection (Dal Pozzolo Credit Fraud): In this highly imbalanced dataset (0.172% fraud), a classifier $M_2$ had a much higher AUROC (85.3%) than $M_1$ (79.5%). However, in the realistic regime where fraud is rare and FNs are expensive, $M_1$ achieved a higher VOROS (90.1% vs 89.6%), correctly identifying it as the better choice for the business case.

Conclusion and Future Directions

VOROS provides a robust framework for evaluating classifiers under uncertainty. It effectively resolves the "crossing curves" dilemma where AUROC cannot distinguish between models that dominate in different cost regimes. While currently focused on binary classification with constant per-class costs, future work may extend VOROS to multi-class problems and instance-varying costs (e.g., the specific dollar amount of a fraudulent transaction).

Summary updated

BibTeX
@inproceedings{ratigan2025voros,
  author = {Christopher Ratigan and Lenore Cowen},
  title = {{The VOROS: Lifting ROC Curves to 3D to Summarize Unbalanced Classifier Performance}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i19.34219},
}
Figure