Rethinking Spatio-Temporal Transformer for Traffic Prediction: Multi-Level Multi-View Augmented Learning Framework

Jiaqi Lin, Qianqian Ren, Xingfeng Lv, Yong Liu, Jianguo SunSSRN 2024 · DOI

TL;DR

This paper introduces LVSTformer, a novel Transformer-based framework for traffic prediction that tackles the complex spatio-temporal dependencies in traffic data. By incorporating multi-level spatial views (local, global, pivotal) and a gated temporal attention mechanism, LVSTformer achieves state-of-the-art performance on several benchmark datasets, significantly improving prediction accuracy.

What Problem Does This Paper Address?

Accurately predicting traffic flow is crucial for intelligent transportation systems, enabling better traffic management, route planning, and congestion avoidance. The core challenge lies in capturing the intricate spatio-temporal correlations within traffic data. Traditional methods like RNNs struggle with spatial dependencies, while CNN-based approaches have limitations with non-Euclidean data. Spatio-Temporal Graph Neural Networks (STGNNs) have shown promise but often falter in modeling long-term temporal dependencies. More recent Transformer-based models, while effective in general time-series forecasting, haven't been specifically optimized for the unique spatio-temporal characteristics of traffic data, particularly multi-scale spatial relationships and temporal periodicity. Furthermore, attention mechanisms can suffer from imbalance, leading to over-reliance on certain features and reduced generalization.

Methods

The LVSTformer framework is built upon several key innovations:

  • Spatio-Temporal Data Embedding Layer: This layer acts as a pre-processing step, aggregating raw traffic data, temporal periodic features (day of week, time of day), and spatial information (graph Laplacian) into a unified representation.
  • Multi-view Spatial Self-Attention (MVSA): This is a core component that addresses multi-level spatial correlations. It comprises three parallel self-attention mechanisms:
  • Local Geographic Self-Attention (LGSA): Captures dependencies between geographically adjacent nodes.
  • Global Semantic Self-Attention (GSSA): Identifies similarities between nodes in functionally similar but potentially distant regions.
  • Pivotal Nodes Self-Attention (PNSA): Focuses on "pivotal" nodes that play a critical role in aggregating and distributing traffic flow, assigning them higher importance.
  • Gated Temporal Self-Attention: To better capture both short- and long-term temporal dependencies and periodicity, this module uses a gating mechanism to dynamically regulate information flow, enhancing the model's ability to learn trends.
  • Spatio-Temporal Context Broadcasting (STCB): This module is introduced to mitigate attention imbalance. It inserts uniform attention between layers to ensure a more balanced distribution of attention scores, preventing over-reliance on specific features and improving generalization.

Datasets

The paper evaluates LVSTformer on six well-known traffic datasets:

  • PeMS03, PeMS04, PeMS07, PeMS08: These are traffic flow datasets from California, USA, varying in the number of sensors and time periods.
  • PeMS-BAY, METR-LA: These are traffic speed datasets from the San Francisco Bay Area and Los Angeles, California, respectively.

No new benchmarks were introduced; the focus was on applying the proposed model to existing, widely used datasets.

Key Insights and Ideas

  • Multi-level spatial dependencies are crucial: The paper strongly argues that simply considering spatial correlations as a whole is insufficient. Differentiating between local, global, and the influence of pivotal nodes leads to a more nuanced and effective understanding of traffic dynamics.
  • Pivotal nodes matter: Identifying and giving more weight to nodes that act as traffic hubs significantly improves prediction accuracy, as these nodes are central to network behavior.
  • Gated temporal attention enhances long-term forecasting: The proposed gated mechanism effectively captures both short-term fluctuations and long-term trends, addressing a key limitation of previous STGNNs.
  • Attention imbalance is a real problem: The STCB module's effectiveness in improving robustness and generalization by ensuring a balanced attention distribution highlights a subtle but important issue in attention-based models.
  • Transformers can be tailored for traffic: While Transformers are powerful, this work demonstrates that specific architectural modifications (multi-view spatial attention, gated temporal attention) are necessary to unlock their full potential for traffic prediction.

Key Results

LVSTformer consistently achieved state-of-the-art performance across the evaluated datasets.

  • Overall Performance: The model achieved the best performance in terms of MAE, RMSE, and MAPE on most datasets, with improvements of up to 4.32% in MAE compared to competing baselines.
  • Long-Term Prediction: LVSTformer showed a widening performance gap over baseline methods as the prediction horizon extended, indicating strong capabilities in long-term forecasting.
  • Ablation Studies: Removing individual components (GCN instead of MVSA, gated temporal attention, different spatial views) led to significant performance degradation, underscoring the importance of each module.
  • Computational Efficiency: LVSTformer demonstrated a favorable balance between accuracy and computational cost, requiring fewer parameters and less training time compared to some advanced baselines while achieving superior results.

Limitations and Future Work

  • Limitations: The paper acknowledges that while STGNNs and attention mechanisms have advanced traffic prediction, they still face challenges. Specifically, STGNNs can accumulate errors in multi-step predictions, and some attention-based methods may overlook valuable information due to attention imbalance. The paper's STCB module aims to address the latter.
  • Future Work: The authors plan to apply LVSTformer to other spatio-temporal prediction tasks beyond traffic prediction, suggesting its potential applicability in broader domains.

Summary updated

BibTeX
@misc{lin2024rethinking,
  author = {Jiaqi Lin and Qianqian Ren and Xingfeng Lv and Yong Liu and Jianguo Sun},
  title = {{Rethinking Spatio-Temporal Transformer for Traffic Prediction: Multi-Level Multi-View Augmented Learning Framework}},
  year = {2024},
  doi = {10.2139/ssrn.4858193},
  howpublished = {SSRN},
}
Figure