Spatiotemporal Residual Regularization with Dynamic Mixtures for Traffic Forecasting

Seongjin Choi, Nicolas Saunier, Martin Trépanier, Lijun SunarXiv 2022 · arXiv · OpenReview

TL;DR

This paper introduces a novel dynamic mixture model to capture the complex, time-varying correlations in traffic forecasting errors. By modeling these errors as a mixture of Gaussian distributions with full covariance, the method significantly improves prediction accuracy and provides interpretable insights into traffic dynamics, outperforming traditional MSE-based approaches.

What Problem Does This Paper Address?

Traffic forecasting models, especially those using deep learning, often rely on simplified assumptions about prediction errors. Typically, errors are assumed to be independent and follow simple distributions like Gaussian or Laplacian. However, real-world traffic data exhibits complex spatiotemporal correlations that change over time. These correlations mean that errors at one sensor or time step are related to errors at others, and this relationship isn't static. Ignoring these dynamic, correlated errors can lead to biased and less accurate forecasts. The challenge is to model these intricate error distributions in a way that is both accurate and computationally feasible.

Methods

The core of this paper is a "dynamic mixture of zero-mean Gaussian distributions" for modeling the forecasting errors. Instead of assuming simple error distributions, they propose a model where the error distribution is a weighted sum of several Gaussian distributions. Each Gaussian component has a full covariance matrix, allowing it to capture complex spatiotemporal correlations.

Key innovations include:

  • Dynamic Mixture Weights: The weights of these Gaussian components are not fixed but change over time, allowing the model to adapt to evolving traffic patterns.
  • Scalable Covariance Parameterization: To handle the high dimensionality of spatiotemporal covariance matrices (which can be very large), they decompose each component's covariance matrix into a Kronecker product of a spatial covariance matrix and a temporal covariance matrix. This significantly reduces computational complexity.
  • Cholesky Factorization for Stability: To ensure stable training and avoid numerical issues like divergence or exploding gradients, they parameterize the precision (inverse covariance) matrices using Cholesky factorization. This allows for efficient computation of determinants and avoids direct matrix inversion during training.
  • Integration with Existing Models: The proposed method is designed as an "add-on" to existing deep learning frameworks, requiring only a few additional parameters to be learned.

Datasets

The authors evaluated their method on several widely-used traffic speed and flow datasets:

  • PEMS-BAY-2017-SPEED: Collected from 325 sensors in the Bay Area, covering 5 months of data.
  • METR-LA: Collected from 207 sensors in Los Angeles County, covering 4 months of data.
  • PEMS-BAY-2022-FLOW: A new dataset for traffic flow.
  • PEMS-BAY-2022-SPEED: A new dataset for traffic speed.

All datasets use a 5-minute resolution. They used Graph Wavenet (GWN) and Spatio-Temporal Graph Convolutional Networks (STGCN) as baseline models.

Key Insights and Ideas

  • Unrealistic Assumptions of MSE/MAE: The paper strongly argues that the standard assumptions behind MSE and MAE loss functions (independent, isotropic errors) are fundamentally flawed for real-world traffic forecasting.
  • Time-Varying Nature of Traffic Errors: The empirical analysis clearly shows that the covariance matrices of traffic errors are not static; they exhibit distinct patterns at different times of the day, highlighting the need for dynamic modeling.
  • Interpretability through Mixture Components: The learned spatial and temporal covariance matrices for each mixture component offer valuable insights. For instance, one component might represent "peak-hour patterns" with strong spatial and temporal correlations, while another might represent "off-peak patterns" with more independent spatial behavior.
  • Scalability is Crucial: The Kronecker product decomposition is a critical enabler, making it feasible to model complex spatiotemporal dependencies without prohibitive computational costs.
  • Cholesky Factorization for Robustness: Parameterizing precision matrices via Cholesky factors is key to achieving stable training, preventing common issues like divergence and exploding gradients that plague direct covariance matrix learning.

Key Results

  • Improved Prediction Accuracy: The proposed dynamic mixture loss function consistently improved performance across various metrics (RMSE, MAPE, MAE) for both GWN and STGCN baseline models on all tested datasets.
  • Longer Horizons Benefit Most: Improvements were more pronounced for longer forecasting horizons (e.g., 60 minutes), indicating the model's enhanced ability to capture evolving traffic dynamics over time.
  • MAPE Reduction: Significant improvements in MAPE were observed, which is particularly important for congested traffic states where speeds are low and errors can be heavily penalized.
  • Interpretable Patterns: Figure 3 demonstrates learned spatial and temporal covariance matrices, showing distinct patterns for different mixture components, which can be interpreted as different traffic regimes (e.g., peak vs. off-peak). Figure 3(B) further shows distinct weekday and weekend patterns in the mixture weights.
  • Ablation Studies:
    • Increasing the number of mixture components (K) generally led to further improvements, up to a certain point, suggesting the model's ability to capture multimodal distributions.
  • A balanced combination of MSE and NLL loss (p < 1) often yielded slightly better results than using pure NLL loss, indicating the benefit of retaining some of the direct mean prediction accuracy from MSE.

Limitations and Future Work

  • Multivariate Gaussian Assumption: The paper acknowledges that using multivariate Gaussian distributions as component distributions, while convenient, might not fully capture the complex non-linearities in real-world traffic data. Future work could explore non-Gaussian distributions or more complex probabilistic models.
  • Kronecker Product Assumption: The assumption that the spatiotemporal covariance can be decomposed into separable spatial and temporal components via a Kronecker product might be too restrictive for some datasets. Future research could explore more flexible covariance parameterizations.
  • Formal Convergence Guarantee: The paper lacks a formal theoretical analysis of the method's convergence properties. Establishing such guarantees would increase confidence in its stability and robustness.
  • Alternative Precision Matrix Structures: Exploring Gaussian Markov Random Fields (GMRFs) or directly modeling sparse spatiotemporal precision matrices could offer alternative ways to model dependencies.
  • Broader Applications: The method's potential extends to other domains with complex spatiotemporal correlations, such as wind speed forecasting, climate modeling, and financial markets.

Summary updated

Abstract

Deep learning-based multivariate and multistep-ahead traffic forecasting models are typically trained with the mean squared error (MSE) or mean absolute error (MAE) as the loss function in a sequence-to-sequence setting, simply assuming that the errors follow an independent and isotropic Gaussian or Laplacian distributions. However, such assumptions are often unrealistic for real-world traffic forecasting tasks, where the probabilistic distribution of spatiotemporal forecasting is very complex with strong concurrent correlations across both sensors and forecasting horizons in a time-varying manner. In this paper, we model the time-varying distribution for the matrix-variate error process as a dynamic mixture of zero-mean Gaussian distributions. To achieve efficiency, flexibility, and scalability, we parameterize each mixture component using a matrix normal distribution and allow the mixture weight to change and be predictable over time. The proposed method can be seamlessly integrated into existing deep-learning frameworks with only a few additional parameters to be learned. We evaluate the performance of the proposed method on a traffic speed forecasting task and find that our method not only improves model performance but also provides interpretable spatiotemporal correlation structures.
BibTeX
@misc{choi2022spatiotemporal,
  author = {Seongjin Choi and Nicolas Saunier and Martin Trépanier and Lijun Sun},
  title = {{Spatiotemporal Residual Regularization with Dynamic Mixtures for Traffic Forecasting}},
  year = {2022},
  eprint = {2212.06653},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.06653},
}
Figure