Abstract
With model trustworthiness being crucial for sensitive real-world applications, practitioners are putting more and more focus on improving the uncertainty calibration of deep neural networks. Calibration errors are designed to quantify the reliability of probabilistic predictions but their estimators are usually biased and inconsistent. In this work, we introduce the framework of proper calibration errors, which relates every calibration error to a proper score and provides a respective upper bound with optimal estimation properties. This relationship can be used to reliably quantify the model calibration improvement. We theoretically and empirically demonstrate the shortcomings of commonly used estimators compared to our approach. Due to the wide applicability of proper scores, this gives a natural extension of recalibration beyond classification.
Introduction
Modern deep neural networks are incredibly accurate, but they often suffer from systematic overconfidence. In sensitive domains like medicine or climate forecasting, it is essential that models are calibrated—meaning their predicted probabilities reflect actual frequencies. For instance, if a model predicts a 70% probability of an event, that event should occur approximately 70% of the time. While various recalibration methods exist, the metrics used to evaluate them, such as the Expected Calibration Error (ECE), are fundamentally flawed. Their estimators are often biased and inconsistent, yielding results that vary significantly based on test set size or data binning strategies. This makes it nearly impossible to reliably quantify if a model has actually become more trustworthy or if the metric is simply noisy. This paper addresses these issues by introducing a robust framework based on proper scores.
Methodology: Proper Calibration Errors
The authors bridge the gap between calibration errors (which measure reliability) and proper scores (which measure overall predictive performance, like the Brier Score). A score is "proper" if the best way to achieve a good score is to predict the true probability distribution.
- Taxonomy of Errors: The paper provides a theoretical hierarchy showing how various existing calibration metrics relate to one another, proving that many common metrics are actually lower bounds of the "true" calibration error.
- Proper Calibration Errors: They define a new class of errors derived from proper scores.
- Root Calibration Upper Bound (RBS): Since the true calibration error is often impossible to estimate without bias, the authors propose using a calibration upper bound (UBS). Specifically, they highlight the square root of the Brier Score (RBS) as a robust estimator. Unlike ECE, RBS has a minimum-variance unbiased estimator, making it stable across different test set sizes.
- Injective Recalibration: They show that for "injective" (one-to-one) recalibration methods—like Temperature Scaling (TS)—their proposed upper bound reliably tracks the real improvement in calibration.
Experimental Evaluation
The authors evaluated their approach across a wide variety of standard computer vision benchmarks (CIFAR-10, CIFAR-100, ImageNet) and regression tasks using architectures like ResNet and DenseNet.
As shown in Figure 1, RBS was the only metric that robustly estimated calibration improvement regardless of test set size. Other metrics often suggested "negative" improvement (making the model worse) on small datasets when the model was actually improving, or showed artificial trends as the sample size increased.
Analysis of Estimator Bias
A key insight of this work is the "ECE Bias Trap." The paper proves that the ECE estimator is strictly convex and monotonically decreasing with respect to data size. This means that with a small test set, ECE will systematically overestimate the error, making it look like the model is worse than it is—or making recalibration look more effective than it actually is.
While common estimators like ECE or KDE-based metrics fluctuate wildly and exhibit high bias in low-data regimes, the proposed RBS remains flat and unbiased. This stability is crucial for practitioners who need to verify model reliability with limited test data.
Conclusion and Future Work
This framework provides a theoretically sound and practically robust way to measure calibration. By leveraging proper scores, it naturally extends recalibration to regression tasks, where calibrating predicted variance is crucial. For example, on the Residential Building dataset, the authors showed that using their framework to calibrate a Mixture Density Network resulted in predicted variances that much more closely matched actual errors.
Future work includes extending this to "Proper U-scores," which could allow for even more sophisticated unbiased estimators, such as those based on the Kernel Calibration Error (KCE). While limitations exist—such as the requirement for injective recalibration functions for certain guarantees—the proposed RBS offers a significant step forward in making uncertainty quantification more reliable for real-world applications.
Summary updated