Conformalized Quantile Regression

Yaniv Romano, Evan Patterson, Emmanuel J. CandèsNeurIPS 2019 · OpenReview

Overview

Conformalized Quantile Regression (CQR) is a powerful framework that combines the flexibility of quantile regression with the rigorous mathematical guarantees of conformal prediction. It allows practitioners to generate prediction intervals that are guaranteed to contain the true value with a user-specified probability, while remaining "adaptive"—meaning the intervals dynamically widen in noisy regions of the data and narrow in stable ones.

The Challenge of Reliable Prediction Intervals

In high-stakes machine learning applications, such as assessing drug efficacy or credit risk, a single point prediction is rarely sufficient; we need to quantify how much to trust that prediction. This is typically done using prediction intervals—a range $[low, high]$ that should contain the true value $Y$ with a specific confidence level (e.g., 90%).

Historically, two main approaches have been used, but both have significant flaws:

  1. Conformal Prediction (CP): CP provides a "distribution-free" guarantee, meaning it works regardless of the underlying data distribution. However, standard CP usually produces intervals of a fixed width across the entire input space. If your data is "heteroscedastic" (noisier in some areas than others), a fixed-width interval is dangerously narrow in noisy spots and unnecessarily wide in quiet ones.
  2. Quantile Regression (QR): QR directly estimates percentiles (e.g., the 5th and 95th) to create adaptive intervals. While highly flexible, QR offers no finite-sample guarantees. In practice, especially with complex models like deep neural networks, QR often "undercovers," providing overly optimistic intervals that miss the true value more often than the target confidence level allows.
Comparison of prediction intervals: Split, Local, and CQR methods.

As illustrated above, standard split conformal prediction (a) yields rigid, fixed-width intervals. Local methods (b) attempt to adapt but can be erratic. CQR (c) successfully adapts to the data's varying noise levels, producing intervals that are tight where possible and wide where necessary, resulting in a lower average length (d) without sacrificing coverage.

The CQR Solution

CQR bridges this gap by using conformal prediction to "calibrate" quantile regression. The authors propose a "Split CQR" algorithm that operates in three main steps:

  1. Training: The data is split into a training set and a calibration set. The training set is used to fit any quantile regression model (such as a Quantile Random Forest or a Deep Neural Network) to estimate the lower ($\alpha/2$) and upper ($1 - \alpha/2$) quantiles.
  2. Calibration: On the held-out calibration set, a conformity score is calculated for each point. This score measures the error of the predicted interval. If the true value falls inside the interval, the score is negative; if it falls outside, the score is positive.
  3. Adjustment: The algorithm identifies a specific quantile of these error scores to serve as a "correction factor." The predicted intervals are then shifted by this factor.

This final adjustment is the secret sauce: it ensures that the resulting intervals satisfy a rigorous coverage guarantee, meaning the probability that a new data point falls within the interval is at least $1-\alpha$.

Key Insights and Results

The authors evaluated CQR across simulated datasets (designed to be highly heteroscedastic) and 11 real-world benchmarks, utilizing both Random Forests and Deep Neural Networks.

  • Adaptivity Equals Efficiency: By allowing the interval width to vary based on the input features, CQR produces intervals that are, on average, significantly shorter than standard conformal methods while maintaining the same safety guarantees. For instance, on a protein structure dataset ("bio"), CQR Random Forests achieved an average interval length of 1.13 compared to 2.09 for standard split conformal methods.
  • Reliable Coverage: While "raw" Quantile Neural Networks frequently failed to reach the target 90% coverage in experiments, the CQR-wrapped versions hit the 90% mark reliably.
  • Model Agnosticism: CQR acts as a wrapper. Practitioners do not need to alter how they train their underlying neural networks or random forests; they simply apply the CQR calibration step post-training.
  • Distribution-Free Guarantees: The mathematical guarantee holds even if the data is non-Gaussian or exhibits heavy tails, provided the data points are "exchangeable" (a slightly weaker assumption than being independent and identically distributed).

Limitations and Future Directions

While highly effective, CQR does have a few limitations. The basic "Split CQR" requires setting aside a portion of data purely for calibration, which can be costly for very small datasets (though the authors note a "cross-conformal" variant can mitigate this). Additionally, in some models, the predicted lower quantile can occasionally cross above the upper quantile. While simple post-processing can fix this, more integrated solutions remain an area for future research. Finally, the authors suggest the underlying logic of CQR could eventually be extended to estimate entire predictive distributions rather than just single intervals.

Summary updated

Abstract

Conformal prediction is a technique for constructing prediction intervals that attain valid coverage in finite samples, without making distributional assumptions. Despite this appeal, existing conformal methods can be unnecessarily conservative because they form intervals of constant or weakly varying length across the input space. In this paper we propose a new method that is fully adaptive to heteroscedasticity. It combines conformal prediction with classical quantile regression, inheriting the advantages of both. We establish a theoretical guarantee of valid coverage, supplemented by extensive experiments on popular regression datasets. We compare the efficiency of conformalized quantile regression to other conformal methods, showing that our method tends to produce shorter intervals.
BibTeX
@inproceedings{romano2019conformalized,
  author = {Yaniv Romano and Evan Patterson and Emmanuel J. Candès},
  title = {{Conformalized Quantile Regression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=oq5hcj5EJS},
}
Figure