High-dimensional multivariate forecasting with low-rank Gaussian Copula Processes

David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, Jan Gasthaus

Introduction

In fields ranging from finance and retail to traffic management and anomaly detection, time series are rarely independent. A surge in one stock often correlates with another, just as a promotion on one product might cannibalize the sales of a competitor. To make accurate probabilistic forecasts, models must account for these underlying dependencies.

However, traditional multivariate models, such as Vector Autoregression or MGARCH, struggle as the number of series ($N$) grows. The number of parameters in a covariance matrix scales at $O(N^2)$, meaning that modeling 1,000 series requires estimating nearly a million relationship parameters. Consequently, most existing high-dimensional models simply assume series are independent, leading to poor risk assessment and missed joint patterns.

This paper introduces a scalable framework for forecasting thousands of correlated time series simultaneously. By combining Recurrent Neural Networks (RNNs) with a low-rank Gaussian copula process, the authors bypass the "curse of dimensionality" that typically makes estimating large covariance matrices computationally impossible, achieving state-of-the-art accuracy with significantly fewer parameters.

Methodology

The authors propose a model that elegantly decomposes the forecasting task into three distinct components:

1. Temporal Dynamics (The RNN)

An LSTM-RNN is utilized to capture the time-series patterns. Crucially, the parameters of this RNN are shared across all series. This allows the model to learn "global" patterns across the entire dataset while maintaining a local state for each individual series to capture its unique trajectory.

2. Marginal Distributions (The Copula)

Real-world data is rarely normally distributed. To handle heavy tails and varying scales, the authors employ a Gaussian Copula. They estimate the marginal distribution of each series non-parametrically using an empirical Cumulative Distribution Function (CDF). This step "warps" the data into a standard normal space where correlations are much easier to compute and manage.

3. Low-Rank Covariance Structure

To solve the $O(N^2)$ parameter bottleneck, the covariance matrix is parameterized as a low-rank-plus-diagonal structure ($D + VV^T$). Instead of learning every possible pair-wise correlation, the model learns a small number of latent "factors" ($r$) that explain the majority of the variance.

Time-varying correlation matrix estimated by the model.

This reduces the parameter complexity from $O(N^2)$ to $O(N \times r)$, where $r$ is typically very small (e.g., 10). This structure allows the model to efficiently estimate complex, time-varying covariance matrices.

Key Insights and Ideas

  • Decoupling Scale from Dependency: By using the Copula approach, the model handles series with vastly different magnitudes (e.g., a highly popular product versus a niche one) without the dependency structure being dominated by the high-magnitude series.
  • Dimension Mini-batching: Because the model uses a low-rank structure and shared parameters, it can be trained by sampling random subsets of dimensions in each iteration. This allows the model to scale to dimensions that would otherwise exceed a GPU's memory limits.
  • Learning the Topology: The model does not just predict values; it predicts how the relationships between series evolve over time. For example, on a dataset of NYC Taxi traffic, the model successfully "learned" the spatial layout of Manhattan. It showed higher correlations between geographically close locations, even though it was never provided with a map.
Learned spatial correlations in the NYC Taxi dataset.

Experimental Results

The model was evaluated on six diverse, real-world datasets ranging from small to very high dimensionality, including Exchange Rates, Solar Production, Electricity Consumption, Traffic Occupancy, NYC Taxi patterns, and Wikipedia Page Views.

  • Accuracy: The proposed "GP-Copula" model outperformed state-of-the-art baselines (including independent RNNs and full-rank multivariate models) on almost all datasets. It improved the Continuous Ranked Probability Score (CRPS)—a measure of distribution accuracy—by 10% to 40% over the next best models.
  • Parameter Efficiency: On the Traffic dataset, the GP-Copula model used only 44,000 parameters to estimate the covariance, a staggering reduction compared to the 38 million parameters required for a standard full-rank model.
  • Stability: The use of the Copula transformation made training much more stable compared to traditional "mean-scaling" methods, which often struggle with the high variance found in real-world data.

Limitations and Future Work

While highly effective, the current implementation assumes that the marginal distributions (the "shape" of the individual series) are stationary over time. If a series has a strong linear trend, the model requires pre-processing, such as de-trending or differencing, to work effectively. Additionally, the rank $r$ is a hyperparameter that must be chosen manually; while $r=10$ worked well for these datasets, highly complex systems might require an automated way to determine the optimal number of latent factors.

Ultimately, this stable estimation of time-varying covariance matrices opens new doors for better anomaly detection, risk assessment, and automated graph analysis in large-scale systems.

Summary updated

Abstract

Predicting the dependencies between observations from multiple time series is critical for applications such as anomaly detection, financial risk management, causal analysis, or demand forecasting. However, the computational and numerical difficulties of estimating time-varying and high-dimensional covariance matrices often limits existing methods to handling at most a few hundred dimensions or requires making strong assumptions on the dependence between series. We propose to combine an RNN-based time series model with a Gaussian copula process output model with a low-rank covariance structure to reduce the computational complexity and handle non-Gaussian marginal distributions. This permits to drastically reduce the number of parameters and consequently allows the modeling of time-varying correlations of thousands of time series. We show on several real-world datasets that our method provides significant accuracy improvements over state-of-the-art baselines and perform an ablation study analyzing the contributions of the different components of our model.
BibTeX
@inproceedings{salinas2019high,
  author = {David Salinas and Michael Bohlke-Schneider and Laurent Callot and Roberto Medico and Jan Gasthaus},
  title = {{High-dimensional multivariate forecasting with low-rank Gaussian Copula Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=F9juRXbm-6},
}
Figure