Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting

Kashif Rasul, Calvin Seward, Ingmar Schuster, Roland Vollgraf

Forecasting a single variable is a well-studied problem, but real-world systems—like city-wide traffic, electricity grids, or stock markets—involve thousands of correlated variables. Traditional methods like Vector Autoregression (VAR) struggle to scale to these dimensions. Modern deep learning approaches often fall into two traps: they either assume variables are independent (missing crucial correlations) or assume the data follows a restrictive distribution like a Multivariate Gaussian. The latter requires estimating a covariance matrix that grows quadratically ($O(D^2)$) and costs $O(D^3)$ to compute, making it impractical for thousands of variables. Furthermore, real-world data is often "messy" and doesn't fit neatly into a Gaussian bell curve.

To address this, researchers introduced TimeGrad, a model that brings the power of denoising diffusion models to multivariate time series forecasting. By combining an autoregressive Recurrent Neural Network (RNN) with a diffusion-based generative head, TimeGrad captures complex, high-dimensional dependencies between variables without relying on oversimplifying assumptions.

How TimeGrad Works

TimeGrad decomposes the forecasting task into two main components: temporal dynamics and a spatial diffusion head.

TimeGrad Architecture
  1. Temporal Dynamics (The RNN): An LSTM or GRU processes the history of the time series along with current covariates (like the day of the week) to produce a hidden state $h_{t-1}$. This state acts as a summary of everything the model knows about the past.
  2. The Diffusion Head: Instead of predicting a simple mean and variance, the model uses a Denoising Diffusion Probabilistic Model (DDPM).
  3. Forward Process: During training, the model takes the real data and gradually adds Gaussian noise over $N$ steps until it becomes pure white noise.
  4. Reverse Process: The model learns a neural network ($\epsilon_\theta$) that tries to predict and remove that noise step-by-step to recover the original data. Crucially, this denoising network is conditioned on the RNN's hidden state.
  5. Architecture: The denoising network uses a series of 1D dilated convolutions to handle the multivariate input efficiently.

At inference time, the model starts with a sample of pure noise and refines it through the learned reverse process to generate a realistic forecast for the next time step. This forecast is then fed back into the RNN to predict the subsequent step.

Key Insights

  • Flexibility over Tractability: Unlike Normalizing Flows, which require specific mathematical structures (like a tractable Jacobian) to remain invertible, diffusion models have almost no restrictions on their functional form. This allows them to learn much more complex, multi-modal distributions.
  • Handling Disconnected Modes: Flow-based models often struggle to model "disconnected" clusters in data because they must apply a continuous transformation. Diffusion models bypass this limitation, making them more robust for complex real-world data.
  • The Diffusion Step Trade-off: An ablation study showed that while $N=100$ steps yielded the best results, the model performed surprisingly well even with only $N=10$ steps. This suggests that for some applications, practitioners can trade a bit of accuracy for significantly faster sampling.

State-of-the-Art Results

The model was evaluated on six standard real-world benchmarks ranging from low-dimensional (Exchange, 8 variables) to high-dimensional (Wikipedia, 2,000 variables). TimeGrad achieved State-of-the-Art (SOTA) results on 5 out of the 6 benchmarks using the Continuous Ranked Probability Score (CRPS), a metric that measures how well a model captures the entire probability distribution rather than just a point estimate.

  • High-Dimensional Success: On the Wikipedia dataset, TimeGrad achieved a CRPS of 0.0485, significantly outperforming the previous best model (Transformer MAF) at 0.063.
  • Traffic Modeling: On the Traffic dataset (963 dimensions), it achieved a CRPS of 0.044, nearly 20% better than the Transformer-based baseline.
Qualitative Forecasting Results

Qualitatively, the model successfully captures sharp daily peaks and provides well-calibrated uncertainty intervals across wildly different scales, as seen in the traffic data predictions.

Limitations and Future Directions

While powerful, TimeGrad has a few limitations. Because the model must iterate $N$ times through the denoising network for every single time step, it is computationally slower at inference time than one-shot models. Exploring non-Markovian diffusion processes could help speed this up. Additionally, the current formulation assumes continuous data; for discrete counts, the model relies on dequantization (adding a small amount of noise). Explicitly modeling discrete distributions could be a fruitful future path. Finally, while this paper used an RNN for the temporal component, replacing it with a Transformer for better long-range dependency modeling, or a Graph Neural Network (GNN) for known spatial relationships, offers exciting avenues for future architectural upgrades.

Summary updated

Abstract

In this work, we propose TimeGrad, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradie...
BibTeX
@inproceedings{rasul2021autoregressive,
  author = {Kashif Rasul and Calvin Seward and Ingmar Schuster and Roland Vollgraf},
  title = {{Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=n9hAgbxgQ84},
}
Figure