The curious case of the test set AUROC

Michael Roberts, Alon Hazan, Sören Dittmer, James H. F. Rudd, Carola-Bibiane Schönlieb

TL;DR

The Area Under the Receiver Operating Characteristic curve (AUROC) is the industry standard for evaluating machine learning classifiers, but it can be dangerously misleading. This paper demonstrates how identical AUROC scores can mask massive distribution shifts between validation and test sets, and proposes a new suite of metrics—including robustness to noise and Wasserstein distance matrices—to better assess a model's real-world generalizability.

What Problem Does This Paper Address?

In the last decade, machine learning models have exploded in complexity, yet our methods for evaluating them have remained stagnant. Most researchers rely on the AUROC to report performance on a test set or use a fixed "optimal" threshold derived from validation data.

The authors argue that this is a "blunt" approach. The fundamental mathematical property of the ROC curve is that it is invariant to any strictly monotonic transformation of model outputs. This means two models can produce identical ROC curves even if their underlying prediction distributions are completely different. In a clinical or engineering setting, a threshold that works perfectly on validation data might fail catastrophically on test data due to a hidden distribution shift, even if the AUROC remains high.

Figure 1. Example validation and test set model output distributions, ROC curves coloured by threshold.

Methods

The paper introduces two classes of "discrepancy scores" to supplement standard AUROC reporting:

  1. Single Dataset Discrepancy Scores: These evaluate a model's robustness in isolation.
  2. Robustness to Bias: Measures how much bias must be added to a class before the distributions become indistinguishable.
  3. Robustness to Noise: Assesses how performance degrades when predictions are diffused with Gaussian noise. This identifies models where the "operating point" is highly sensitive to small perturbations.
  4. Multiple Data Cohort Discrepancies: These compare performance between validation and test sets.
  5. Drift in Sensitivity and Specificity: Calculates the Euclidean distance between the (sensitivity, specificity) pairs of the two cohorts across all possible thresholds.
  6. Wasserstein Distance Matrix: Uses the 2-Wasserstein ("earth-mover") distance to quantify how much the positive and negative class distributions move between datasets.

Datasets

The paper primarily uses synthetic "example" distributions to illustrate the mathematical pitfalls of the AUROC. It does not benchmark against specific large-scale public datasets (like ImageNet or MNIST), but rather provides a framework and a public codebase intended for researchers to apply to their own domain-specific datasets.

Key Insights and Ideas

  • The Monotonicity Trap: Because the ROC curve only cares about the rank of predictions, it ignores the actual values. A model that predicts values clustered tightly around 0.5 and a model that predicts values polarized at 0.1 and 0.9 can have the exact same AUROC.
  • Threshold Fragility: A "good" test AUROC does not guarantee that a threshold chosen during validation will remain optimal. If the model outputs shift (e.g., due to different data sources), the previously chosen operating point may no longer provide the desired balance of sensitivity and specificity.
  • Visualizing the "Drift": By coloring ROC curves by their threshold values (as seen in Figure 1), researchers can visually identify when the same point on the curve corresponds to wildly different thresholds in different cohorts.

Key Results

The authors demonstrate a striking case where a validation set and a test set both yield an AUROC of 0.92, yet their output distributions are fundamentally different (see Figure 1).

To quantify these differences, they introduce the Drift Score. In their example, they show how sensitivity and specificity can drift erratically across different thresholds, providing a much more granular view of model failure than a single AUROC number.

Figure 2. The robustness to bias and noise for the model outputs on the example validation and test data.

They also advocate for the use of a Wasserstein distance matrix. Ideally, a model should have large distances between positive and negative classes within a cohort (demarcation) but small distances for the same class across different cohorts (consistency).

Limitations and Future Work

The authors acknowledge that the metrics they propose are not exhaustive and may not be the "best" possible approaches for every scenario. They frame this manuscript as a "call to arms" for the machine learning community to move beyond the AUROC.

Future work involves a more systematic and concerted effort to define robust evaluation standards that account for distribution shifts in real-world deployment scenarios, particularly in high-stakes fields like medicine.

Summary updated

Abstract

Whilst the size and complexity of ML models have rapidly and significantly increased over the past decade, the methods for assessing their performance have not kept pace. In particular, among the many potential performance metrics, the ML community stubbornly continues to use (a) the area under the receiver operating characteristic curve (AUROC) for a validation and test cohort (distinct from training data) or (b) the sensitivity and specificity for the test data at an optimal threshold determined from the validation ROC. However, we argue that considering scores derived from the test ROC curve alone gives only a narrow insight into how a model performs and its ability to generalise.
BibTeX
@article{roberts2024curious,
  author = {Michael Roberts and Alon Hazan and Sören Dittmer and James H. F. Rudd and Carola-Bibiane Schönlieb},
  title = {{The curious case of the test set AUROC}},
  journal = {Nat. Mac. Intell.},
  year = {2024},
  doi = {10.1038/s42256-024-00817-7},
}
Figure