Surrogate uncertainty estimation for your time series forecasting black-box: learn when to trust

Leonid Erlygin, Vladimir Zholobov, Valeriia Baklanova, Evgeny Sokolovskiy, Alexey Zaytsev

Abstract

Machine learning models play a vital role in time series forecasting. These models, however, often overlook an important element: point uncertainty estimates. Incorporating these estimates is crucial for effective risk management, informed model selection, and decision-making. To address this issue, our research introduces a method for uncertainty estimation. We employ a surrogate Gaussian process regression model. It enhances any base regression model with reasonable uncertainty estimates. This approach stands out for its computational efficiency. It only necessitates training one supplementary surrogate and avoids any data-specific assumptions. Furthermore, this method requires only the presence of the base model as a black box and its respective training data. The effectiveness of our approach is supported by experimental results. Using various time-series forecasting data, we found that our surrogate model-based technique delivers significantly more accurate confidence intervals. These techniques outperform both bootstrap-based and built-in methods in a medium-data regime. This superiority holds across a range of base model types, including linear regression, ARIMA, gradient boosting, and neural networks.

Introduction

In high-stakes fields like finance, medicine, or autonomous driving, a point forecast (e.g., "the stock price will be $100 tomorrow") is insufficient. Decision-makers require uncertainty quantification: is that estimate a high-confidence prediction, or is there a wide range of possible outcomes? While some models provide built-in uncertainty estimates, many popular "black-box" tools—such as Gradient Boosting (CatBoost/XGBoost), ARIMA, or complex Neural Networks—often lack them or provide poorly calibrated intervals.

The standard remedy is the bootstrap method, which involves training the same model dozens of times on different data slices. However, this is computationally expensive and frequently fails to account for the sequential dependencies inherent in time series data.

Methodology: The BAMOES Framework

We introduce BAMOES (Base-model-enhanced surrogate model). Instead of retraining the original "base" model, BAMOES trains a single Gaussian Process (GP) to act as a surrogate.

Dual-Purpose Loss Function

Standard surrogates typically attempt to fit only the original training data. BAMOES employs a more sophisticated loss function with two primary components:

  1. Data Likelihood: It fits the original training data to capture inherent noise (aleatoric uncertainty).
  2. Model Mimicry: It samples a set of "supplemental points" ($X'$) across the input space and queries the black-box base model for its predictions. The surrogate is penalized if its predictions deviate from the black box.

By observing how the black box behaves even in regions without real data, the GP surrogate learns to mimic the base model's logic while providing mathematically rigorous uncertainty estimates.

Experimental Evaluation

BAMOES was evaluated against a robust set of benchmarks:

  • TSForecasting (Monash Archive): 26 public datasets covering tourism, energy, and banking.
  • M4 Dataset: Testing against state-of-the-art neural networks like PatchTST.
  • Financial Data: A case study involving market volatility during the COVID-19 pandemic to test out-of-distribution performance.

Key Results

The primary metric for evaluation was the Miscalibration Area, which measures how much predicted confidence intervals deviate from actual error rates (lower is better).

Comparison of BAMOES OLS and Bootstrap OLS: (a) Time series forecasting with uncertainty intervals; (b) Calibration plot showing miscalibration area.

As shown in the figure above, BAMOES provides tighter, more accurate calibration compared to traditional bootstrap methods. Across OLS, ARIMA, and CatBoost base models, BAMOES consistently achieved the lowest mean ranks for miscalibration.

Critical difference diagram for miscalibration area across multiple benchmarks, showing BAMOES OLS as the top-performing method.

Statistical analysis via critical difference diagrams confirms that BAMOES significantly outperforms sophisticated time-series bootstrap methods (like BSAP BS) in medium-data regimes. Furthermore, when applied to the PatchTST neural network, BAMOES provided superior uncertainty estimates compared to the popular Monte-Carlo (MC) Dropout technique.

Conclusion and Future Work

BAMOES offers a computationally efficient alternative to ensembles, requiring only one supplementary surrogate training run ($O(N^3)$ complexity). While standard GPs can struggle with extremely large datasets, the use of sparse Gaussian Processes remains a promising avenue for scaling. Future research will explore expanding this surrogate approach to Large Language Models (LLMs) and other NLP tasks where uncertainty quantification remains a critical challenge.

Summary updated

Abstract

Machine learning models play a vital role in time series forecasting. These models, however, often overlook an important element: point uncertainty estimates. Incorporating these estimates is crucial for effective risk management, informed model selection, and decision-making.To address this issue, our research introduces a method for uncertainty estimation. We employ a surrogate Gaussian process regression model. It enhances any base regression model with reasonable uncertainty estimates. This approach stands out for its computational efficiency. It only necessitates training one supplementary surrogate and avoids any data-specific assumptions. Furthermore, this method for work requires only the presence of the base model as a black box and its respective training data. The effectiveness of our approach is supported by experimental results. Using various time-series forecasting data, we found that our surrogate model-based technique delivers significantly more accurate confidence intervals. These techniques outperform both bootstrap-based and built-in methods in a medium-data regime. This superiority holds across a range of base model types, including a linear regression, ARIMA, gradient boosting and a neural network.
BibTeX
@inproceedings{erlygin2023surrogate,
  author = {Leonid Erlygin and Vladimir Zholobov and Valeriia Baklanova and Evgeny Sokolovskiy and Alexey Zaytsev},
  title = {{Surrogate uncertainty estimation for your time series forecasting black-box: learn when to trust}},
  booktitle = {Workshops @ ICDM},
  year = {2023},
  doi = {10.1109/icdmw60847.2023.00162},
}
Figure