The curious case of the test set AUROC

Michael Roberts, Alon Hazan, Sören Dittmer, James H. F. Rudd, Carola-Bibiane Schönlieb

Abstract

While the Area Under the Receiver Operating Characteristic curve (AUROC) is the industry standard for evaluating classifiers, it is dangerously blind to distribution shifts. This paper demonstrates how a model can maintain a high test AUROC while becoming practically useless in production due to prediction drift. We propose a new suite of metrics to measure prediction discrepancy and threshold stability, moving beyond rank-based evaluation to ensure real-world reliability.

Introduction: The Problem with AUROC

In machine learning, a high AUROC on a held-out test set is often considered the gold-standard proof of a model’s success. However, this single scalar value provides only a narrow window into a model's real-world generalizability. The core issue is that AUROC is rank-based: it measures a model's ability to distinguish between positive and negative cases but ignores the actual values of the predictions.

If a model’s output distribution shifts significantly between the validation set and the test set—a common occurrence in clinical or real-world settings—the AUROC might remain high, but the "optimal threshold" chosen during validation will fail. Consequently, a model can be "accurate" in its rankings but completely uncalibrated for deployment.

Proposed Methodology

To address these limitations, we introduce two categories of supplementary scores designed to move beyond single scalar values:

1. Single Dataset Discrepancy Scores

These evaluate a model's robustness in isolation:

  • Robustness to Bias: Measures the amount of "shift" (bias) that must be added to a class before distributions overlap and performance degrades.
  • Robustness to Noise: Quantifies sensitivity to small perturbations (Gaussian noise) in model outputs.

2. Multiple Data Cohort Discrepancies

These compare behavior between validation and test sets:

  • Drift in Sensitivity and Specificity ($D_{sens/spec}$): Calculates the Euclidean distance between the sensitivity/specificity of the validation and test sets across all possible thresholds.
  • Wasserstein Distance Matrix: Uses the "earth-mover" distance to quantify how much the distributions of positive and negative classes have moved relative to each other across different datasets.

Key Insights

  • The Monotonicity Trap: AUROC is invariant to any strictly monotonic function. Shifting every prediction value upward leaves the AUROC unchanged, even though it breaks the classification threshold.
  • The Moving Target of "Optimal" Thresholds: Practitioners typically select a threshold on the validation set. Distribution shifts in the test set can move this "optimal" point into regions of the ROC curve with drastically different sensitivity and specificity.
  • Coherence vs. Distinction: A robust model should maintain "coherence" (alignment) across different data sources. The Wasserstein distance matrix helps visualize if a "Positive" prediction in one dataset represents the same underlying reality in another.

Key Results

Identical AUROC, Different Reality

We demonstrate scenarios where both validation and test sets yield an identical AUROC (e.g., 0.92), yet the underlying prediction distributions differ drastically. As shown below, applying a validation-derived threshold to the test set can result in nearly zero sensitivity, rendering the model useless despite its high AUROC.

Comparison of prediction distributions and ROC curves for validation and test sets.

Quantifying Fragility

Using the "Robustness to Bias" metric, we show that models appearing stable on the surface can be extremely fragile. A "Relative AUROC" can plummet with even a minor perturbation, revealing hidden vulnerabilities that standard metrics miss.

Robustness to bias and noise perturbations.

Limitations and Future Work

These proposed scores are a starting point and are not exhaustive. While this work focuses on the ROC curve, the same arguments apply to Precision-Recall curves. Future research is required to establish "acceptable" ranges for these metrics across various domains, such as medical imaging versus natural language processing. This paper serves as a call to arms for the ML community to develop more robust, distribution-aware evaluation standards.

Summary updated

BibTeX
@article{roberts2024curious,
  author = {Michael Roberts and Alon Hazan and Sören Dittmer and James H. F. Rudd and Carola-Bibiane Schönlieb},
  title = {{The curious case of the test set AUROC}},
  journal = {Nat. Mac. Intell.},
  year = {2024},
  doi = {10.1038/s42256-024-00817-7},
}
Figure