Abstract
In this work, we propose \texttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this end, we use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest through a Markov chain using Langevin sampling. We demonstrate experimentally that the proposed autoregressive denoising diffusion model is the new state-of-the-art multivariate probabilistic forecasting method on real-world data sets with thousands of correlated dimensions. We hope that this method is a useful tool for practitioners and lays the foundation for future research in this area.
Introduction
Predicting the future of multiple related variables—such as traffic across hundreds of sensors or electricity demand for numerous households—is a critical challenge in modern decision-making. Multivariate probabilistic forecasting is particularly difficult due to two primary factors:
- High Dimensionality: As the number of variables ($D$) increases, modeling their joint distribution becomes computationally expensive, often scaling at $O(D^3)$ for traditional Gaussian methods.
- Complex Dependencies: Real-world data frequently exhibits non-linear relationships and "heavy tails" that standard linear models (e.g., VAR) or simple deep learning architectures fail to capture accurately.
While existing solutions like Normalizing Flows are powerful, they often require restrictive architectures to remain mathematically tractable. Energy-Based Models (EBMs) offer more flexibility but are notoriously difficult to train.
Methodology: The TimeGrad Model
The proposed model, TimeGrad, treats forecasting as a conditional generative problem by combining the temporal reasoning of Recurrent Neural Networks (RNNs) with the generative power of diffusion models.
Core Components
- Autoregressive RNN: An LSTM or GRU processes historical time series data and covariates (e.g., day of the week) to maintain a hidden state ($h_{t-1}$). This state serves as a summary of all past information.
- Diffusion Probabilistic Model: Instead of predicting a single value, the model learns to "denoise" data. During training, a forward process gradually adds Gaussian noise to a real data point until it becomes white noise. The model then learns a neural network ($\epsilon_\theta$) to predict the noise added at any given step.
- Inference via Reverse Process: At inference time, the model starts with pure noise and iteratively removes it—conditioned on the RNN's hidden state—to sample a realistic future value.
TimeGrad is one of the first models to successfully adapt diffusion processes for autoregressive time series. It utilizes a WaveNet-style architecture with dilated 1D convolutions for the denoising network, allowing it to capture local patterns across different dimensions of the multivariate vector.
Experimental Evaluation
The model was evaluated on six diverse, real-world benchmarks ranging from low-dimensional (8 currencies in Exchange) to high-dimensional (2,000 pages in Wikipedia). Other datasets included Solar, Electricity, Traffic, and Taxi.
Key Insights
- EBM Flexibility: Diffusion models provide the flexibility of Energy-Based Models but are trained using a stable, regression-like loss.
- Dynamic Conditioning: By conditioning the diffusion process on the RNN hidden state, the model learns a time-varying distribution where the "shape" of the potential future evolves based on past events.
- Scaling Robustness: Scaling each time series by its context window mean proved crucial for handling datasets with variables of vastly different magnitudes.
Results
TimeGrad achieved state-of-the-art (SOTA) performance on nearly all benchmarks, as measured by the Continuous Ranked Probability Score (CRPS). It significantly outperformed competitive baselines like Transformer-MAF and GP-Copula, particularly on high-dimensional datasets like Taxi and Wikipedia.
An important finding from the ablation studies is the efficiency of the diffusion steps. While the model was trained with $N=100$ steps, it can produce high-quality forecasts with as few as $N=10$ steps during inference, providing a viable path toward faster sampling in production environments.
Conclusion and Future Work
TimeGrad establishes a new benchmark for multivariate probabilistic forecasting. Future research directions include:
- Inference Speed: Exploring non-Markovian diffusion processes (e.g., DDIM) to reduce the computational cost of iterative sampling.
- Discrete Data: Adapting the model for discrete counts without relying on dequantization.
- Architectural Evolution: Replacing the RNN with a Transformer to better capture long-term historical dependencies.
Summary updated