Calibration tests in multi-class classification: A unifying framework

David Widmann, Fredrik Lindsten, Dave Zachariah

Abstract

Assessing the calibration of multi-class probabilistic classifiers is critical for safety-sensitive applications. Traditional metrics like Expected Calibration Error (ECE) often rely on binning, which suffers from bias and poor scalability. This work introduces a unifying framework based on vector-valued Reproducing Kernel Hilbert Spaces (RKHS) to define the Squared Kernel Calibration Error (SKCE). We provide unbiased estimators for SKCE with known asymptotic distributions, enabling principled statistical tests for multi-class calibration. Experiments on synthetic data and modern neural networks demonstrate that our framework provides more reliable assessments than standard top-1 binning approaches.

Introduction

In many machine learning applications, particularly in medicine or autonomous driving, the reliability of a model's uncertainty estimates is as important as its accuracy. A well-calibrated model's predicted probabilities should reflect the true frequency of outcomes. While binary calibration is well-understood, multi-class calibration presents significant challenges. Most existing methods focus solely on the "top-1" prediction, ignoring the rest of the probability distribution. Furthermore, standard metrics like Expected Calibration Error (ECE) rely on partitioning the probability space into bins, a process that introduces statistical bias and scales poorly with the number of classes.

Methods: A Kernel-Based Framework

To address these limitations, we propose a general framework for calibration error based on vector-valued Reproducing Kernel Hilbert Spaces (RKHS) and matrix-valued kernels. This approach avoids binning by using kernels to measure the discrepancy between predicted probability distributions and actual outcomes.

Within this framework, we derive the Squared Kernel Calibration Error (SKCE). We show that many existing measures, including ECE and Maximum Mean Calibration Error (MMCE), are special cases of this unified theory. Crucially, we develop three estimators for SKCE with different computational and statistical properties:

Comparison of SKCE estimators: notation, definition, properties, and computational complexity.

The unbiased estimators ($\widehat{\text{SKCE}}_{\text{uq}}$ and $\widehat{\text{SKCE}}_{\text{ul}}$) are particularly valuable because their asymptotic distributions are known, allowing them to be used as test statistics. This transforms abstract calibration scores into interpretable p-values for hypothesis testing.

Experimental Evaluation

The framework was validated using both synthetic generative models and modern deep neural networks.

Synthetic Models

We designed three synthetic models (M1, M2, M3) to simulate various degrees of miscalibration across different numbers of classes ($m=2$ to $1000$). These controlled experiments allowed us to verify the theoretical properties of our estimators.

Distribution of calibration error estimates for synthetic models M1, M2, and M3 across different estimators. The vertical dashed lines indicate the true calibration error.

As shown in the distributions above, the standard binned ECE and the biased SKCE ($\widehat{\text{SKCE}}_{\text{b}}$) exhibit significant bias, especially as the number of classes increases. In contrast, the proposed unbiased estimators provide centered and consistent estimates of the true calibration error.

Modern Neural Networks

We applied our calibration tests to a variety of popular architectures (ResNet, DenseNet, VGG, etc.) trained on CIFAR-10. While these models often appear reasonably calibrated under "top-1" metrics, our stronger multi-class tests yielded p-values near zero for almost all models. This indicates that their full output distributions are statistically unreliable, highlighting the importance of moving beyond top-1 calibration.

Discussion and Conclusion

By framing calibration as a statistical test within an RKHS framework, we provide a more rigorous and interpretable way to evaluate model uncertainty. Our method avoids the pitfalls of binning and scales effectively to high-dimensional probability simplices.

Future work will focus on the impact of kernel selection and hyperparameter tuning, as the power of these tests depends on the choice of the matrix-valued kernel. Additionally, while the asymptotic p-values are practical for large datasets, developing tighter distribution-free bounds remains an open challenge. To support the community, we have released our implementation in the Julia packages CalibrationErrors.jl and CalibrationTests.jl.

Summary updated

BibTeX
@inproceedings{widmann2019calibration,
  author = {David Widmann and Fredrik Lindsten and Dave Zachariah},
  title = {{Calibration tests in multi-class classification: A unifying framework}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=C3u6uZttmHB},
}
Figure