Abstract
We extend conformal prediction methodology beyond the case of exchangeable data. In particular, we show that a weighted version of conformal prediction can be used to compute distribution-free prediction intervals for problems in which the test and training covariate distributions differ, but the likelihood ratio between the two distributions is known---or, in practice, can be estimated accurately from a set of unlabeled data (test covariate points). Our weighted extension of conformal prediction also applies more broadly, to settings in which the data satisfies a certain weighted notion of exchangeability. We discuss other potential applications of our new conformal methodology, including latent variable and missing data problems.
Introduction
In machine learning, we often assume that training and test data are drawn from the same distribution, a property known as exchangeability. Under this assumption, conformal prediction is a powerful tool that allows us to wrap any predictive model in a "band" guaranteed to contain the true value with a user-defined probability (e.g., 90%).
However, real-world data frequently exhibits covariate shift, where the distribution of features ($X$) changes between training and deployment. In these scenarios, standard conformal prediction often "under-covers," producing prediction intervals that are too narrow and fail to meet the promised confidence level. This paper introduces a weighted extension to the conformal framework to restore these critical reliability guarantees.
Methodology: Weighted Conformal Prediction
The proposed method, Weighted Conformal Prediction, accounts for shifts by re-weighting nonconformity scores using the likelihood ratio $w(x) = d\tilde{P}_X(x) / dP_X(x)$. This ratio represents how much more likely a feature vector $x$ is in the test set compared to the training set.
Instead of treating every training point equally, the algorithm constructs a weighted empirical distribution of scores. Points that resemble the test data more closely receive higher weights. The prediction interval is then derived from the quantile of this weighted distribution. This approach is theoretically supported by a new concept of weighted exchangeability, ensuring validity even when the i.i.d. assumption is violated.
Empirical Results
The authors demonstrated the method using the UCI Airfoil Self-Noise dataset, simulating a covariate shift via exponential tilting.
As shown in Figure 1, the standard method fails significantly under shift. By contrast, the weighted approach successfully restores coverage. While using "oracle" weights (the true ratio) yielded 90.8% coverage, the method remains highly effective even when the ratio must be estimated.
As illustrated in Figure 2, estimating the likelihood ratio using standard classifiers like logistic regression or random forests provides nearly perfect coverage (91.0%), making the method highly practical for real-world pipelines where the true shift is unknown.
Discussion and Limitations
While weighted conformal prediction restores distribution-free robustness, it introduces a trade-off in interval dispersion. Weighting effectively reduces the "effective sample size," which can lead to more variable intervals. Furthermore, the method's success depends on the quality of the likelihood ratio estimate, which is challenging in high-dimensional spaces.
Future work remains in achieving "local" conditional coverage (valid for a specific $x_0$) more efficiently and exploring applications in missing data and latent variable modeling. Despite these challenges, weighted conformal prediction represents a significant step toward reliable machine learning in non-stationary environments.
Summary updated