Conformal Prediction Under Covariate Shift

Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas

TL;DR

This paper introduces a novel "weighted conformal prediction" method that extends traditional conformal prediction to handle situations where the distribution of test data differs from the training data (covariate shift). This allows for reliable prediction intervals even when the data generating process changes, a common problem in real-world applications.

What Problem Does This Paper Address?

Traditional conformal prediction methods rely on the assumption that training and test data are exchangeable, meaning they are drawn from the same underlying distribution and their order doesn't matter. This assumption is often violated in practice. For instance, a model trained on historical data might perform poorly on new data collected under different conditions (e.g., a change in sensor calibration, user behavior, or environmental factors). This phenomenon is known as covariate shift. When this shift occurs, standard conformal prediction intervals can become unreliable, often undercovering the true values. The paper addresses this critical gap by developing a method that can maintain valid coverage guarantees even in the presence of covariate shift, provided the ratio of the test to training covariate distributions is known or can be estimated.

Methods

The core innovation is a weighted conformal prediction framework. Instead of treating all training data points equally, this method assigns weights to each nonconformity score (a measure of how "unusual" a data point is) based on the likelihood ratio of the test covariate distribution to the training covariate distribution. This weighting effectively makes the training data "look like" the test data, allowing standard conformal prediction machinery to be applied.

Specifically, the authors:

  • Define weighted exchangeability: A generalization of exchangeability that accounts for varying weights.
  • Develop a weighted quantile lemma: A theoretical result that forms the basis for the weighted conformal prediction.
  • Propose a weighted conformal prediction algorithm: This algorithm uses the weighted nonconformity scores to construct prediction intervals.
  • Introduce weighted split conformal prediction: A computationally more efficient variant that uses a separate validation set to pre-fit a model, making it practical for large datasets.

The key idea is that by weighting the nonconformity scores, the empirical distribution of these scores from the training data better approximates the distribution of scores for the test data, thus restoring the validity of the prediction intervals.

Datasets

The paper uses the airfoil dataset from the UCI Machine Learning Repository for empirical evaluation. This dataset contains measurements related to airfoil performance. The authors create synthetic covariate shift by randomly partitioning the data and then applying an "exponential tilting" to the covariate distribution of a subset of the data to simulate the shift. They do not introduce new benchmark datasets.

Key Insights and Ideas

  • Covariate shift breaks standard conformal prediction: The paper clearly demonstrates that traditional conformal prediction methods fail to provide valid coverage when covariate shift is present, leading to undercoverage.
  • Likelihood ratio is key: The ability to know or estimate the likelihood ratio between the test and training covariate distributions is crucial for the success of the weighted method. This ratio acts as a bridge to correct for the distribution mismatch.
  • Weighting is a powerful correction: Assigning weights proportional to the likelihood ratio effectively "re-weights" the training data to resemble the test data, restoring the validity of conformal prediction.
  • Effective sample size reduction: While weighted conformal prediction restores coverage, it can lead to a reduction in the "effective sample size," potentially increasing the width of the prediction intervals. This is an inherent trade-off for achieving valid coverage under shift.
  • Estimated weights can work: The paper shows empirically that using estimated weights (derived from logistic regression or random forests) can achieve good coverage, even when the true likelihood ratio is unknown.

Key Results

  • Empirical demonstration of undercoverage: For the airfoil dataset, ordinary split conformal prediction showed an average coverage of 82.2% on the shifted test set, significantly below the nominal 90%.
  • Restored coverage with oracle weights: Using the true (oracle) likelihood ratio, weighted split conformal prediction achieved an average coverage of 90.8%, bringing it back to the desired level.
  • Good coverage with estimated weights: Using weights estimated via logistic regression and random forests also yielded good average coverages (91.0% and 91.0% respectively), demonstrating the practical applicability of the method.
  • Trade-off in interval width: Weighted conformal prediction intervals were observed to be longer than unweighted ones when accounting for effective sample size, highlighting the trade-off between coverage validity and interval width.

Limitations and Future Work

  • Computational complexity: While split conformal prediction offers efficiency, the general weighted conformal prediction can be computationally intensive, especially when dealing with complex score functions or high-dimensional data.
  • Unknown likelihood ratio: The method relies on knowing or estimating the likelihood ratio. While estimation is shown to be effective, its accuracy can impact performance.
  • Effective sample size: The reduction in effective sample size can lead to wider prediction intervals, which might be undesirable in some applications.
  • Future work: The authors suggest exploring applications in graphical models with covariate shift and missing covariates. They also touch upon the challenge of achieving approximate conditional coverage, which remains an open area. The paper also notes that the upper bound on coverage in Lemma 1 and Theorem 1 might not be achievable without further conditions on the weight functions in the weighted case, unlike the unweighted case.

Summary updated

Abstract

We extend conformal prediction methodology beyond the case of exchangeable data. In particular, we show that a weighted version of conformal prediction can be used to compute distribution-free prediction intervals for problems in which the test and training covariate distributions differ, but the likelihood ratio between these two distributions is known---or, in practice, can be estimated accurately with access to a large set of unlabeled data (test covariate points). Our weighted extension of conformal prediction also applies more generally, to settings in which the data satisfies a certain weighted notion of exchangeability. We discuss other potential applications of our new conformal methodology, including latent variable and missing data problems.
BibTeX
@inproceedings{tibshirani2019conformal,
  author = {Ryan J. Tibshirani and Rina Foygel Barber and Emmanuel J. Candès and Aaditya Ramdas},
  title = {{Conformal Prediction Under Covariate Shift}},
  booktitle = {NeurIPS},
  year = {2019},
  url = {https://openreview.net/forum?id=5QOJFrDhfiR},
}
Figure