Modular Conformal Calibration

Charles Marx, Shengjia Zhao, Willie Neiswanger, Stefano Ermon

In high-stakes fields such as medicine or autonomous navigation, knowing how much to trust a model is just as important as the prediction itself. However, modern high-performance models, like deep neural networks, are notorious for being uncalibrated—they might exhibit 90% confidence while only being accurate 60% of the time.

While recalibration methods exist to turn uncalibrated models into calibrated ones using held-out data, they often suffer from two major flaws. First, they typically rely on restrictive assumptions, requiring the base model to already be probabilistic (e.g., outputting a mean and variance). Second, they struggle with the "sharpness trade-off": a model can be perfectly calibrated by simply outputting the global average of the data, but such a model is uninformative. The ideal model produces distributions that are both calibrated and tightly centered around the truth.

To address these challenges, researchers have introduced a unified, "plug-and-play" framework called Modular Conformal Calibration (MCC), which transforms any regression model into a well-calibrated probabilistic predictor.

The Modular Conformal Calibration (MCC) Framework

The core innovation of MCC is breaking the recalibration process into three distinct, modular components. This design allows practitioners to transform any regression model—even non-probabilistic ones—into a calibrated probabilistic model.

The three components of the Modular Conformal Calibration framework: Base Predictor, Calibration Score, and Interpolation Algorithm.

The three "Lego blocks" of the framework are:

  1. The Base Predictor ($f$): This can be any model, such as a point estimator, an interval predictor, or a quantile regressor. It does not need to be probabilistic out of the box.
  2. The Calibration Score ($\phi$): A function that measures the error or "offness" of a prediction. For a point model, this might be the simple residue ($y - \hat{y}$). For a distribution model, it could be the predicted CDF value or a z-score.
  3. The Interpolation Algorithm ($\psi$): This function maps the calibration scores from a held-out dataset to the unit interval $[0, 1]$ to produce a final cumulative distribution function (CDF).

By choosing specific combinations of these blocks, the MCC framework recovers well-known existing methods. For example, combining a distribution base predictor, a CDF score, and linear interpolation yields Isotonic Calibration. Swapping linear interpolation for a randomized one results in Conformal Calibration.

To improve upon these standard combinations, the authors introduce Neural Autoregressive Flow (NAF) interpolation. NAF uses a small neural network to ensure the resulting calibrated distribution is smooth and differentiable, which is crucial for calculating log-likelihoods in downstream tasks.

Experimental Validation and Key Insights

The MCC framework was evaluated across 17 tabular regression datasets (primarily from the UCI Machine Learning Repository), covering diverse tasks like housing price prediction, concrete strength estimation, and blog post popularity.

Unification and Theoretical Guarantees

By framing recalibration as a modular process, the authors provide the first finite-sample calibration guarantees for Isotonic Calibration—a method that has been used for decades without such formal proofs. Furthermore, they proved that the intervals produced by MCC's calibrated distributions are mathematically comparable to those produced by Conformal Interval Prediction, meaning practitioners do not lose theoretical rigor by moving from simple intervals to full distributions.

Quantiles are King for Sharpness

One of the most surprising empirical findings is the effectiveness of quantile regression. Training a base model to predict quantiles (e.g., the 10th, 20th, ..., 90th percentiles) and then recalibrating it leads to much sharper and more accurate distributions than starting with a standard Gaussian (mean/variance) model.

Violin plots demonstrating that quantile-based predictors consistently yield sharper (narrower) intervals compared to standard point or distribution models.

New algorithms designed within the MCC framework—specifically those using quantile base predictors—outperformed existing baselines in terms of sharpness (measured by the width of confidence intervals) and proper scoring rules like the Continuous Ranked Probability Score (CRPS). Across all 17 datasets, MCC algorithms achieved near-perfect calibration, with an average Expected Calibration Error (ECE) of less than 0.007.

The Smoothness vs. Guarantees Trade-off

The experiments also highlighted a fundamental trade-off in interpolation. Randomized interpolation provides "perfect" theoretical calibration but produces discontinuous, step-like distributions. Conversely, NAF interpolation provides smooth, highly usable distributions that, while lacking the absolute perfect theoretical guarantee of randomized methods, remain nearly perfectly calibrated in practice.

Limitations and Future Work

While MCC offers a robust and flexible approach to recalibration, it has a few limitations:

  • Tie-Breaking: The theoretical guarantees rely on calibration scores being "absolutely continuous" (i.e., no two samples share the exact same score). While generally true in continuous regression, this can pose issues with low-precision or heavily discretized data.
  • Computational Overhead: Utilizing NAF interpolation is computationally heavier than simple linear interpolation, as it requires training a small auxiliary neural network during the recalibration phase.
  • High-Dimensional Outputs: The current framework is designed for scalar regression. Extending this modular approach to multi-variate outputs or structured data modalities (like images or text) remains an open challenge for future research.

Summary updated

Abstract

Uncertainty estimates must be calibrated (i.e., accurate) and sharp (i.e., informative) in order to be useful. This has motivated a variety of methods for recalibration, which use held-out data to turn an uncalibrated model into a calibrated model. However, the applicability of existing methods is limited due to their assumption that the original model is also a probabilistic model. We introduce a versatile class of algorithms for recalibration in regression that we call modular conformal calibration (MCC). This framework allows one to transform any regression model into a calibrated probabilistic model. The modular design of MCC allows us to make simple adjustments to existing algorithms that enable well-behaved distribution predictions. We also provide finite-sample calibration guarantees for MCC algorithms. Our framework recovers isotonic recalibration, conformal calibration, and conformal interval prediction, implying that our theoretical results apply to those methods as well. Finally, we conduct an empirical study of MCC on 17 regression datasets. Our results show that new algorithms designed in our framework achieve near-perfect calibration and improve sharpness relative to existing methods.
BibTeX
@inproceedings{marx2022modular,
  author = {Charles Marx and Shengjia Zhao and Willie Neiswanger and Stefano Ermon},
  title = {{Modular Conformal Calibration}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=yeQxlC3rHI},
}
Figure