Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs

Gerardo Flores, Alyssa H. Smith, Julia A Fukuyama, Ashia C. Wilson

TL;DR

This paper introduces a new evaluation framework for machine learning models in clinical settings that goes beyond traditional metrics like accuracy and AUC-ROC. It accounts for crucial clinical priorities: calibration, robustness to changing data distributions (label shift), and the varying costs of different types of errors. This leads to more reliable and clinically relevant model assessments.

What Problem Does This Paper Address?

In clinical decision support, machine learning models often output probabilistic scores to guide diagnoses and prognoses. However, standard evaluation metrics like accuracy and AUC-ROC fail to adequately capture what's important in a clinical context. These metrics often:

  • Ignore calibration: They don't check if the predicted probabilities actually reflect the true likelihood of an event. A model might be good at ranking but bad at giving reliable probability estimates.
  • Are sensitive to data shifts: Models trained on one dataset (e.g., a balanced research dataset) may perform poorly when deployed in a real-world setting with different class prevalences (label shift).
  • Don't account for asymmetric error costs: In medicine, the cost of a false positive (e.g., unnecessary treatment) is often very different from the cost of a false negative (e.g., missed diagnosis). Standard metrics treat these errors equally.

This misalignment means that models that perform well on standard metrics might not translate to optimal clinical decisions, potentially leading to suboptimal patient care. The authors aim to bridge this gap by proposing evaluation methods that directly incorporate these clinical priorities.

Methods

The paper proposes a principled evaluation framework built on the theory of proper scoring rules, specifically leveraging the Schervish representation. This representation shows that any proper scoring rule can be viewed as an average of cost-weighted errors. The authors extend this by:

  • Characterizing label shift uncertainty: They propose using lower and upper bounds on class balance to represent uncertainty about deployment-time prevalences.
  • Averaging cost-sensitive metrics: They develop methods to average performance metrics (like net benefit) over this range of class balances, using a log-odds transformation for a more natural averaging.
  • Adapting existing metrics: They derive adjusted versions of common metrics like accuracy and net benefit that are robust to label shift and incorporate asymmetric costs. This includes a "DCA Log Score" which isolates miscalibration over clinically relevant probability intervals.

Datasets

The paper uses the eICU dataset (specifically, a publicly available subset) for evaluating subgroup performance in predicting in-hospital mortality using APACHE IV scores.

Key Insights and Ideas

  • AUC-ROC is not a panacea: While better than accuracy for handling varying class balances, AUC-ROC completely disregards calibration and doesn't allow for independent specification of label shift and error costs, making it difficult to interpret in clinical contexts.
  • Calibration matters for decision-making: Even if a model isn't perfectly calibrated globally, ensuring calibration within a specific "threshold band" relevant to clinical decisions can be sufficient for optimal decision-making.
  • Causal framing is key: By adopting a causal perspective (D → Y → X for classification), the authors can better understand and address label shift and its impact on model evaluation.
  • Decomposition is powerful: The proposed framework allows for the decomposition of performance differences into components attributable to label shift, calibration, and sharpness, providing a clearer understanding of why models differ in performance.

Key Results

The paper introduces several new evaluation metrics and frameworks, including:

  • Prior-Adjusted Maximum Accuracy (PAMA) and Prior-Adjusted Maximum Net Benefit (PAMNB): These metrics account for label shift.
  • DCA Log Score: A metric that measures miscalibration in cost-aligned units over relevant probability intervals.
  • Decomposition of performance differences: Applied to the eICU dataset, the authors demonstrate how their framework can disentangle the effects of label shift, calibration, and sharpness on subgroup performance disparities. For example, in the eICU data, they show that performance differences between racial groups are largely driven by label shift and calibration issues, rather than fundamental differences in model sharpness.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Cost Uncertainty: While their DCA log score handles uncertain cost ratios, the resulting dilogarithmic expressions are analytically complex and computationally intensive. Future work could explore approximations for better interpretability and scalability.
  • Sampling Variability under Label Shift: Estimating uncertainty for metrics involving asymmetric costs and label shift is challenging due to high variance and potential bias.
  • Adaptive Base Rate Estimation: The current framework assumes known deployment class prevalences. Future work could incorporate uncertainty in prevalence estimation and its impact on predictions.
  • Asymmetric Cost Parameterization: The semantics of varying cost ratios and their impact on averaging properties need further theoretical exploration to provide robust guidelines for parameter selection.

Summary updated

BibTeX
@inproceedings{flores2025aligning,
  author = {Gerardo Flores and Alyssa H. Smith and Julia A Fukuyama and Ashia C. Wilson},
  title = {{Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=b7Ka31HvXU},
}
Figure