Conformal prediction interval for dynamic time-series

Chen Xu, Yao XieICML 2021 · OpenReview

In fields ranging from renewable energy forecasting to supply chain management, point predictions are rarely sufficient. Decision-makers need to know how much to trust a model's forecast, a requirement typically addressed by constructing prediction intervals.

The gold standard for generating "distribution-free" intervals—meaning they don't rely on strict assumptions about the underlying data distribution—is Conformal Prediction (CP). However, standard CP relies heavily on the assumption of exchangeability, which dictates that the order of the data does not matter. Time-series data fundamentally violates this assumption: observations are temporally correlated, often non-stationary (the underlying rules change over time), and spatially dependent.

Existing workarounds for applying CP to time-series either require "data splitting," which wastes precious data in small-sample regimes, or are computationally ruinous because they require retraining complex models thousands of times. To solve this, researchers have developed EnbPI (Ensemble Batch Prediction Intervals), a novel framework that provides theoretically backed uncertainty quantification adapted to non-stationary data.

How EnbPI Works

EnbPI leverages the power of ensemble learning (such as Random Forests or Bootstrap Aggregating) to build reliable intervals efficiently, without the need for expensive model refitting. The framework operates on three core mechanisms:

  1. Leave-One-Out (LOO) Residuals via Bootstrap: Instead of retraining a model for every single data point, EnbPI uses bootstrap samples. For each training point $i$, it aggregates only the predictions from bootstrap models that did not see point $i$ during training. This provides a "leave-one-out" estimate and a corresponding residual (error) essentially for free.
  2. Sliding Window Calibration: To handle non-stationarity, EnbPI maintains a sliding window of the most recent $T$ residuals. The width of the prediction interval is dynamically determined by the $(1-\alpha)$ quantile of these recent residuals.
  3. Sequential Feedback: As new data arrives, the algorithm updates its "pool" of residuals without necessarily retraining the underlying regression models. This allows the interval to widen or shrink dynamically based on recent performance.

Theoretically, the authors demonstrate that as long as the errors are "strongly mixing" (meaning correlations between distant time points fade away), EnbPI achieves approximately valid marginal coverage, bypassing the strict exchangeability requirement.

Key Insights

  • Computational Efficiency: By utilizing the "out-of-bag" samples from bootstrap training, EnbPI acquires its calibration data without extra training steps. This reduces the computational complexity to $O(B)$ (where $B$ is the number of bootstrap models) rather than $O(BT)$, offering a massive speedup for large datasets.
  • Model Agnosticism: EnbPI acts as a wrapper. Whether the underlying model is Ridge regression, a Random Forest, or an LSTM, EnbPI maintains coverage regardless of the chosen "black box."
  • Adaptability to Change: The sliding window of residuals is a simple but powerful mechanism. If a sudden "change point" occurs (e.g., a rapid weather shift in solar forecasting), the model's recent residuals will spike, prompting EnbPI to automatically widen its intervals to reflect the increased uncertainty.

Experimental Results

The method was rigorously evaluated on several real-world challenges, including solar radiation forecasting, wind energy prediction, and credit card fraud detection.

Superior Coverage: In energy forecasting tasks, EnbPI consistently hit the target coverage (e.g., 90%). In contrast, traditional methods like ARIMA failed significantly as the target confidence increased. While ARIMA sometimes produced narrower intervals, those intervals frequently failed to cover the true value. EnbPI's intervals remained valid and reliable.

Coverage and width comparison between EnbPI and ARIMA.

Data Efficiency: EnbPI significantly outperformed "Split-Conformal" methods (like Inductive Conformal Prediction, or ICP), especially when training data was scarce. Because EnbPI does not require a separate, held-out calibration set, it utilizes 100% of the available data for both training and interval calibration.

Performance comparison across varying amounts of training data.

Limitations and Future Directions

While highly effective, EnbPI has limitations. It guarantees marginal coverage (meaning it is correct 90% of the time on average) but can struggle with conditional coverage at specific, highly volatile times—such as the rapid transition of "noon-time" in solar radiation data. Furthermore, the theoretical guarantees rely on the underlying regression estimator being reasonably consistent; if the base model is fundamentally incapable of learning the pattern, the resulting intervals may become excessively wide.

Future work aims to extend EnbPI to classification problems in computer vision and to develop tighter theoretical bounds for the width of the generated intervals.

Summary updated

Abstract

We develop a method to construct distribution-free prediction intervals for dynamic time-series, called \Verb|EnbPI| that wraps around any bootstrap ensemble estimator to construct sequential predi...
BibTeX
@inproceedings{xu2021conformal,
  author = {Chen Xu and Yao Xie},
  title = {{Conformal prediction interval for dynamic time-series}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=r8z4p1Y60c},
}
Figure