Is this model reliable for everyone? Testing for strong calibration

Jean Feng

Abstract

In a well-calibrated risk prediction model, the average predicted probability is close to the true event rate for any given subgroup. Such models are reliable across heterogeneous populations and satisfy strong notions of algorithmic fairness. However, the task of auditing a model for strong calibration is well-known to be difficult -- particularly for machine learning (ML) algorithms -- due to the sheer number of potential subgroups. As such, common practice is to only assess calibration with respect to a few predefined subgroups. Recent developments in goodness-of-fit testing offer potential solutions but are not designed for settings with weak signal or where the poorly calibrated subgroup is small, as they either overly subdivide the data or fail to divide the data at all. We introduce a new testing procedure based on the following insight: if we can reorder observations by their expected residuals, there should be a change in the association between the predicted and observed residuals along this sequence if a poorly calibrated subgroup exists. This lets us reframe the problem of calibration testing into one of changepoint detection, for which powerful methods already exist. We begin with introducing a sample-splitting procedure where a portion of the data is used to train a suite of candidate models for predicting the residual, and the remaining data are used to perform a score-based cumulative sum (CUSUM) test. To further improve power, we then extend this adaptive CUSUM test to incorporate cross-validation, while maintaining Type I error control under minimal assumptions. Compared to existing methods, the proposed procedure consistently achieved higher power in simulation studies and more than doubled the power when auditing a mortality risk prediction model.

Introduction

Machine learning models often perform well on average but can fail silently on specific, hidden subgroups. When a model outputs a probability, we expect it to be accurate—a property known as calibration. While "moderate calibration" ensures predictions are accurate on average, the ideal standard is strong calibration: predicted probabilities should match true event rates across all possible subgroups.

Verifying strong calibration is notoriously difficult due to the "curse of dimensionality." As features increase, the number of potential subgroups explodes. Existing methods often require massive datasets or lose statistical power by arbitrarily binning data. This paper introduces AdaptScoreCUSUM, a novel statistical test that reframes the search for poorly calibrated subgroups as a one-dimensional changepoint detection problem, allowing for reliable auditing even with limited data.

The AdaptScoreCUSUM Method

The core innovation of this work is shifting the perspective from a high-dimensional search to a 1D changepoint detection task. By sorting data based on expected errors, problematic observations are naturally grouped together, bypassing the need to pre-define specific subgroups.

The three-step AdaptScoreCUSUM procedure for testing strong calibration.

The procedure follows three main steps:

  1. Train Residual Models: A portion of the audit data is used to train a diverse ensemble of "residual models" (e.g., Random Forests, Kernel methods). These models predict the errors (residuals) of the original ML model rather than the outcome itself.
  2. Order the Data: The remaining test data is sorted based on these predicted residuals, from largest to smallest.
  3. CUSUM Test: A score-based Cumulative Sum (CUSUM) test scans the ordered list. If a poorly calibrated subgroup exists, observed errors will accumulate, causing a spike in the cumulative score. A Monte Carlo bootstrap procedure is used to calculate critical thresholds, ensuring strict Type I error control.

Experimental Results

The authors evaluated AdaptScoreCUSUM against several baselines, including the Hosmer-Lemeshow test and multicalibration-based tests, using both simulated and real-world datasets.

Statistical power comparison and variable importance plots across different sample sizes and tolerances.

Key Findings

  • Superior Statistical Power: In simulations, the proposed method consistently outperformed existing baselines, particularly when the miscalibrated subgroup was small. Power was shown to grow significantly with the amount of test data allocated.
  • Hospital Readmission (ZSFG Dataset): Auditing a model for 30-day unplanned hospital readmissions, the test detected that the model overestimated risk for a specific subgroup. Variable Importance (VI) plots identified the number of prior patient contacts as the primary driver of this miscalibration.
  • Toxic Comments (CivilComments Dataset): When auditing a neural network for text toxicity, the test detected widespread under-estimation of toxicity. VI plots highlighted that comments related to Black identities were most critical in defining the poorly calibrated subgroup.

Discussion and Insights

A key insight from this research is that diversity in auditing models is crucial; using the same algorithm for both the original model and the residual model can lead to missed patterns. Furthermore, the mathematical analysis suggests a "phase change" in data splitting: statistical power is maximized by allocating just enough data to train baseline residual models and reserving the majority for the testing phase.

While currently optimized for binary classification, future work aims to extend this framework to regression tasks and explore its application in detecting societal biases, distinguishing them from purely statistical miscalibration.

Summary updated

Abstract

In a well-calibrated risk prediction model, the average predicted probability is close to the true event rate for any given subgroup. Such models are reliable across heterogeneous populations and satisfy strong notions of algorithmic fairness. However, the task of auditing a model for strong calibration is well-known to be difficult -- particularly for machine learning (ML) algorithms -- due to the sheer number of potential subgroups. As such, common practice is to only assess calibration with respect to a few predefined subgroups. Recent developments in goodness-of-fit testing offer potential solutions but are not designed for settings with weak signal or where the poorly calibrated subgroup is small, as they either overly subdivide the data or fail to divide the data at all. We introduce a new testing procedure based on the following insight: if we can reorder observations by their expected residuals, there should be a change in the association between the predicted and observed residuals along this sequence if a poorly calibrated subgroup exists. This lets us reframe the problem of calibration testing into one of changepoint detection, for which powerful methods already exist. We begin with introducing a sample-splitting procedure where a portion of the data is used to train a suite of candidate models for predicting the residual, and the remaining data are used to perform a score-based cumulative sum (CUSUM) test. To further improve power, we then extend this adaptive CUSUM test to incorporate cross-validation, while maintaining Type I error control under minimal assumptions. Compared to existing methods, the proposed procedure consistently achieved higher power in simulation studies and more than doubled the power when auditing a mortality risk prediction model.
BibTeX
@inproceedings{feng2024is,
  author = {Jean Feng},
  title = {{Is this model reliable for everyone? Testing for strong calibration}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=lF2yckFCDW},
}
Figure