Modular Conformal Calibration

Charles Marx, Shengjia Zhao, Willie Neiswanger, Stefano Ermon

Abstract

Uncertainty estimates must be calibrated (i.e., accurate) and sharp (i.e., informative) in order to be useful. This has motivated a variety of methods for recalibration, which use held-out data to turn an uncalibrated model into a calibrated model. However, the applicability of existing methods is limited due to their assumption that the original model is also a probabilistic model. We introduce a versatile class of algorithms for recalibration in regression that we call Modular Conformal Calibration (MCC). This framework allows one to transform any regression model into a calibrated probabilistic model. The modular design of MCC allows us to make simple adjustments to existing algorithms that enable well-behaved distribution predictions. We also provide finite-sample calibration guarantees for MCC algorithms. Our framework recovers isotonic recalibration, conformal calibration, and conformal interval prediction, implying that our theoretical results apply to those methods as well. Finally, we conduct an empirical study of MCC on 17 regression datasets. Our results show that new algorithms designed in our framework achieve near-perfect calibration and improve sharpness relative to existing methods.

Introduction

In machine learning, accuracy isn't everything. For high-stakes applications like medicine or autonomous driving, we need to know how much to trust a prediction. A "calibrated" model is one where a 90% confidence interval actually contains the true value 90% of the time. Unfortunately, modern high-performance models, such as deep neural networks, are notorious for being overconfident or providing no uncertainty estimates at all. While "recalibration" methods exist to fix this using a small held-out dataset, they are often fragmented: some only work if the original model is already probabilistic, while others produce "jumpy," discontinuous distributions that are hard to use in downstream tasks.

The Modular Conformal Calibration Framework

The core of this research is the Modular Conformal Calibration (MCC) framework, which decomposes the recalibration process into three independent, "plug-and-play" modules. This unification allows researchers to transform any regression model—whether it outputs a single point, an interval, or a full distribution—into a calibrated probabilistic model.

The three modules of the MCC framework: Base Predictor, Calibration Score, and Interpolation Algorithm.

As shown in the figure above, the framework consists of:

  1. Base Predictor ($f$): Any model, ranging from point estimates and intervals to ensembles and full distributions.
  2. Calibration Score ($\phi$): A function measuring how "surprising" a true label is relative to the prediction (e.g., residuals for point predictors or CDF values for distributions).
  3. Interpolation Algorithm ($\psi$): The step mapping calibration scores into valid probabilities. Options include simple linear interpolation or sophisticated Neural Autoregressive Flows (NAF) for smooth, differentiable outputs.

The authors prove a finite-sample guarantee: any algorithm built using this framework will have a calibration error that shrinks at a rate of $O(1/n)$, where $n$ is the size of the calibration dataset.

Empirical Study and Results

The researchers conducted an extensive empirical study using 17 tabular regression datasets from the UCI Machine Learning Repository, covering physical, social, and biological domains. They used 60% of the data for training, 20% for calibration, and 20% for testing.

Across all datasets, MCC algorithms achieved near-perfect calibration, with an average Expected Calibration Error (ECE) of less than 0.007. Beyond accuracy, the framework significantly improved "sharpness"—the informativeness of the uncertainty intervals.

Comparison of 95% and 90% interval sizes across various base predictors, demonstrating the sharpness of quantile-based models.

A key insight from the results is that using a Quantile Predictor as the base model often outperformed models specifically trained to output full distributions (like Gaussian estimators). As seen in the violin plots above, quantile-based predictors frequently resulted in narrower, sharper intervals while maintaining calibration. Furthermore, using NAF as an interpolator allowed for smooth distributions that are more useful for downstream tasks requiring gradients or likelihoods.

Conclusion

MCC provides a flexible toolkit for producing uncertainty estimates that are both mathematically guaranteed to be accurate and empirically sharp. By showing that established methods like Isotonic and Conformal Calibration are specific configurations of MCC, the authors provide a unified theory for the field. Future work may involve automated model selection to pick the best MCC components for a given task and extending this modular framework to classification problems.

Summary updated

Abstract

Uncertainty estimates must be calibrated (i.e., accurate) and sharp (i.e., informative) in order to be useful. This has motivated a variety of methods for recalibration, which use held-out data to turn an uncalibrated model into a calibrated model. However, the applicability of existing methods is limited due to their assumption that the original model is also a probabilistic model. We introduce a versatile class of algorithms for recalibration in regression that we call Modular Conformal Calibration (MCC). This framework allows one to transform any regression model into a calibrated probabilistic model. The modular design of MCC allows us to make simple adjustments to existing algorithms that enable well-behaved distribution predictions. We also provide finite-sample calibration guarantees for MCC algorithms. Our framework recovers isotonic recalibration, conformal calibration, and conformal interval prediction, implying that our theoretical results apply to those methods as well. Finally, we conduct an empirical study of MCC on 17 regression datasets. Our results show that new algorithms designed in our framework achieve near-perfect calibration and improve sharpness relative to existing methods.
BibTeX
@inproceedings{marx2022modular,
  author = {Charles Marx and Shengjia Zhao and Willie Neiswanger and Stefano Ermon},
  title = {{Modular Conformal Calibration}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=yeQxlC3rHI},
}
Figure