Better Batch for Deep Probabilistic Time Series Forecasting

Vincent Zhihao Zheng, Seongjin Choi, Lijun Sun

Abstract

Deep probabilistic time series forecasting has gained attention for its ability to provide nonlinear approximation and valuable uncertainty quantification for decision-making. However, existing models often oversimplify the problem by assuming a time-independent error process and overlooking serial correlation. To overcome this limitation, we propose an innovative training method that incorporates error autocorrelation to enhance probabilistic forecasting accuracy. Our method constructs a mini-batch as a collection of $D$ consecutive time series segments for model training. It explicitly learns a time-varying covariance matrix over each mini-batch, encoding error correlation among adjacent time steps. The learned covariance matrix can be used to improve prediction accuracy and enhance uncertainty quantification. We evaluate our method on two different neural forecasting models and multiple public datasets. Experimental results confirm the effectiveness of the proposed approach in improving the performance of both models across a range of datasets, resulting in notable improvements in predictive accuracy.

Introduction

In probabilistic time series forecasting, the objective is to estimate the future distribution of a variable rather than just a single point estimate. Most state-of-the-art deep learning models, such as DeepAR, rely on a univariate Gaussian likelihood, which assumes that prediction errors are independent and identically distributed (i.i.d.) over time.

However, this assumption is frequently violated in real-world applications. If a model fails to capture a specific trend or seasonal component, the resulting errors will exhibit serial correlation, known as autocorrelation. Ignoring this structure leads to poorly calibrated uncertainty intervals and suboptimal predictive performance. This research introduces a "Better Batch" training strategy that explicitly models error autocorrelation using a dynamic, time-varying covariance matrix.

Methodology

The proposed framework moves from a univariate Gaussian likelihood to a multivariate Gaussian likelihood within each training batch to capture temporal dependencies in errors.

The "Better Batch" Structure

Instead of training on isolated, randomly sampled time segments, the model is trained on $D$ consecutive segments. This temporal grouping allows the model to explicitly observe and learn the evolution of errors over a sequence.

The Better Batch structure groups consecutive segments to capture error evolution.

Dynamic Covariance Modeling

The framework learns a time-varying covariance matrix $\Sigma_t$. This matrix is parameterized as a weighted sum of base kernels (specifically Squared Exponential kernels with varying lengthscales) and an identity matrix representing white noise. An auxiliary neural network branch predicts these weights based on the main model's hidden state, allowing the correlation structure to adapt dynamically to the current context of the time series. During prediction, the model uses this learned covariance to "calibrate" its next step based on the residuals observed in previous steps.

Experimental Results

The method was evaluated on two architectures—DeepAR (RNN-based) and a GPT-style Transformer—across a variety of public benchmarks from the GluonTS library, including finance, traffic, and energy datasets.

Predictive Accuracy and Uncertainty

The "Better Batch" approach yielded significant improvements in predictive accuracy, with an average relative reduction in Continuous Ranked Probability Score (CRPS) of 8.80% for DeepAR and 9.76% for the Transformer. The method was particularly effective at improving uncertainty quantification, showing gains of 18-25% on sparse datasets like m1_quarterly. It notably improved 0.9-risk metrics, indicating better performance in quantifying "worst-case" scenarios.

Learned covariance matrix at 16:00 showing structured error correlation during peak hours.

Qualitative Analysis

Visual analysis of the learned covariance matrices confirms that the model successfully captures time-varying patterns. For instance, in traffic data, error correlations are often stronger and more structured during peak activity periods (e.g., 16:00) compared to quieter morning periods (e.g., 06:00).

Conclusion and Future Work

By relaxing the i.i.d. error assumption and explicitly modeling autocorrelation, the "Better Batch" strategy significantly enhances the performance and reliability of deep probabilistic forecasting. Future research will explore more flexible kernel parameterizations, such as learnable Toeplitz matrices, and extend the framework to multivariate models to capture contemporaneous correlations across different time series variables.

Summary updated

Abstract

Deep probabilistic time series forecasting has gained attention for its ability to provide nonlinear approximation and valuable uncertainty quantification for decision-making. However, existing models often oversimplify the problem by assuming a time-independent error process and overlooking serial correlation. To overcome this limitation, we propose an innovative training method that incorporates error autocorrelation to enhance probabilistic forecasting accuracy. Our method constructs a mini-batch as a collection of $D$ consecutive time series segments for model training. It explicitly learns a time-varying covariance matrix over each mini-batch, encoding error correlation among adjacent time steps. The learned covariance matrix can be used to improve prediction accuracy and enhance uncertainty quantification. We evaluate our method on two different neural forecasting models and multiple public datasets. Experimental results confirm the effectiveness of the proposed approach in improving the performance of both models across a range of datasets, resulting in notable improvements in predictive accuracy.
BibTeX
@inproceedings{zheng2024better,
  author = {Vincent Zhihao Zheng and Seongjin Choi and Lijun Sun},
  title = {{Better Batch for Deep Probabilistic Time Series Forecasting}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=Tt2luw7X2q},
}
Figure