Data Efficient Calibration for Conformal Prediction

This direction is split into other directions: data-efficient uncertainty quantification, calibration data overhead. Its last version stays readable and citable here.

Version 1 of 1166 papers194 conceptsfrozen 2026-10-04

Conformal prediction has emerged as a premier framework for uncertainty quantification because it offers a rare promise: distribution-free coverage guarantees. If you want a model to be 95% certain that its prediction set contains the true label, conformal prediction provides a mathematically rigorous way to achieve that, regardless of the model's architecture or the underlying data distribution.

However, this guarantee usually comes with a "data tax." To ensure that the uncertainty estimates are not biased by the model's training process, standard Split Conformal Prediction (SCP) requires partitioning labeled data into two disjoint sets: one for training the model and another for calibration. In data-scarce regimes—such as medical imaging or rare event detection—this split creates a painful trade-off. Every sample used for calibration is a sample lost to the training process, potentially degrading the model's primary performance.

The Geometry of the Data Splitting Penalty

The fundamental tension in conformal inference is between validity (the guarantee that the interval contains the truth) and efficiency (the desire for that interval to be as small as possible).

As Yunzhen Yao and Michael Gastpar demonstrate in Non-Asymptotic Analysis of Efficiency in Conformalized Regression, the width of a prediction interval is sensitive to the sizes of both the training and calibration sets. If the calibration set is too small, the "quantile" used to set the interval boundary becomes noisy, leading to intervals that are unnecessarily wide or highly unstable. Conversely, if the training set is too small, the model's "conformity scores" (its measure of error) will be large and erratic, also resulting in bloated intervals.

Batiste Le Bars and Pierre Humbert argue in On Volume Minimization in Conformal Regression that the learning step is actually more data-hungry than the calibration step. Their work suggests that practitioners should often lean toward allocating more data to training, as a better-optimized model produces more informative scores that naturally lead to tighter intervals, even if the calibration set is slightly smaller.

Recycling Data via Ensembles and Resampling

To bypass the data-splitting penalty, researchers have developed "sample-efficient" variants that allow data to do double duty. The goal is to use the entire dataset for both training and calibration without violating the exchangeability assumptions that make conformal guarantees work.

One powerful approach is the Jackknife+ framework, which uses leave-one-out residuals to calibrate. Mattia Rigotti and colleagues show in Adaptive Conformal Regression with Jackknife+ Rescaled Scores that combining this with score rescaling allows for local adaptivity—intervals that grow and shrink based on local difficulty—even when data is limited.

Similarly, Chirag Gupta and Aaditya Ramdas introduced Quantile Out-of-Bag (QOOB) in Nested conformal prediction and quantile out-of-bag ensemble methods. By leveraging the "out-of-bag" samples already present in ensemble methods like Random Forests, QOOB provides valid calibration without ever setting aside a dedicated hold-out set. This "recycling" of data ensures that every observation contributes to both the model's predictive power and the precision of its uncertainty bounds.

Augmenting the Calibration Pool

When real labeled data is truly exhausted, a new frontier of research suggests looking elsewhere for calibration signals: synthetic data and unlabeled samples.

In Synthetic-powered predictive inference, Meshi Bashari and Yaniv Romano propose using generative models to create "synthetic" calibration sets. By using a "score transporter" to account for the shift between real and synthetic distributions, they can produce much tighter prediction sets than standard methods, effectively using AI-generated data to "subsidize" the uncertainty quantification of a real-world model.

Alternatively, Santiago Mazuelas explores Split conformal classification with unsupervised calibration, demonstrating that it is possible to maintain coverage guarantees using unlabeled data. By matching the distribution of unlabeled test-like data to the training set, this approach removes the requirement for a dedicated labeled calibration set entirely, a massive boon for domains where labels are expensive but raw data is plentiful.

Efficiency through Score Aggregation

Finally, for systems using multiple models, the way we combine uncertainties matters for data efficiency. Traditional methods often merge the final "prediction sets" of different models, which is statistically conservative and leads to large, uninformative regions.

Yash Patel and Ambuj Tewari address this in Conformal Score Aggregation (CSA). By aggregating in "score-space" rather than "set-space," CSA produces significantly smaller uncertainty regions. This efficiency is crucial because it allows ensembles to provide high-precision guarantees without requiring the massive calibration sets that multi-stage geometric methods usually demand.

By optimizing data allocation, recycling samples through resampling, and augmenting calibration with synthetic or unlabeled sources, this research direction transforms conformal prediction from a data-hungry luxury into a practical tool for high-stakes, data-constrained environments.

Papers in this version

Showing 60 of 166 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_data_efficient_calibration_for_conformal_05a0f6d1,
  title = {Data Efficient Calibration for Conformal Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-efficient-calibration-for-conformal-prediction/rdr_05a0f6d15aea43598622b49a7d8a5be3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_data_efficient_calibration_for_conformal_05a0f6d1_v1,
  title = {Data Efficient Calibration for Conformal Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-efficient-calibration-for-conformal-prediction/rdr_05a0f6d15aea43598622b49a7d8a5be3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}