TrafFormer: A Transformer Model for Predicting Long-term Traffic

David Alexander Tedjopurnomo, Farhana Murtaza Choudhury, A. Kai QinarXiv 2023 · OpenReview

TL;DR

This paper introduces TrafFormer, a novel Transformer-based model designed for long-term traffic prediction (up to 24 hours ahead). By leveraging the attention mechanism and incorporating time-day embeddings, TrafFormer overcomes the limitations of traditional recurrent models in capturing complex, long-range traffic patterns, demonstrating superior performance on real-world datasets.

What Problem Does This Paper Address?

Traffic congestion is a significant economic and societal problem, costing billions annually. Accurate traffic prediction is crucial for mitigating this issue, enabling better urban planning and traffic management. However, existing research has predominantly focused on short-term predictions (up to an hour), which are insufficient for proactive planning. Long-term prediction (e.g., 24 hours ahead) is more challenging due to the complex, non-linear, and often unstable nature of traffic patterns over extended periods. Traditional models, often based on recurrent neural networks (RNNs) like LSTMs, struggle with long-term dependencies due to error propagation and the lossy summarization of input sequences. This paper aims to bridge this gap by developing a model capable of reliable long-term traffic forecasting.

Methods

The core of this paper is the introduction of TrafFormer, a modified Transformer model adapted for traffic prediction. Unlike RNNs that process data sequentially, Transformers utilize an attention mechanism that allows the model to weigh the importance of different parts of the input sequence simultaneously. This is particularly beneficial for long-term prediction, as it enables direct access to historical data without the degradation caused by recurrent error propagation.

Key innovations in TrafFormer include:

  • Transformer Architecture: The paper leverages the encoder part of the Transformer, known for its ability to capture long-range dependencies.
  • Spatio-Temporal Multi-Head Attention: This module is designed to simultaneously attend to different time steps and traffic sensor locations, capturing both temporal and spatial correlations in traffic data.
  • Time-Day Embedding: To address the temporal aspect of long-term prediction, TrafFormer incorporates learned embeddings for time-of-day (in 5-minute intervals) and day-of-week. These embeddings capture medium-term and long-term traffic patterns, such as daily and weekly cycles, which are crucial for predicting traffic 24 hours in advance.
  • Hybrid Input: The model utilizes both short-term (past hour) and medium-term (past day) traffic data as input, providing a richer context for prediction.

Datasets

The authors evaluated TrafFormer on two widely used traffic speed datasets:

  • METR-LA: Collected from 207 traffic loop detectors in Los Angeles County, spanning March 2012 to June 2012, with 5-minute granularity.
  • PEMS-BAY: A subset of the Caltrans PEMS dataset, covering 325 traffic loop detectors in the San Francisco Bay Area from January 2017 to May 2017, also with 5-minute granularity.

These datasets were chosen for their common usage in traffic prediction research, ensuring reproducibility and comparability. The paper notes that METR-LA has a higher percentage of missing data compared to PEMS-BAY.

Key Insights and Ideas

  • Transformers are well-suited for long-term temporal dependencies: The paper strongly advocates for Transformers over RNNs for long-term prediction, highlighting how the attention mechanism circumvents the instability and error propagation issues inherent in recurrent structures.
  • Time and day embeddings are critical for long-term patterns: Simply using past traffic speed is insufficient. Explicitly encoding temporal context (time of day, day of week) through learned embeddings is vital for capturing the cyclical and varied patterns that influence traffic over 24 hours.
  • Spatio-temporal attention is key to understanding traffic flow: Traffic at one location is influenced by traffic at neighboring locations and at different times. The spatio-temporal attention mechanism effectively models these interdependencies.
  • Long-term prediction is inherently more challenging: The paper clearly articulates why predicting 24 hours ahead is harder than predicting 1 hour ahead, emphasizing the need for models that can capture short, medium, and long-term traffic patterns.

Key Results

TrafFormer demonstrated superior performance across various prediction horizons (6, 12, 18, and 24 hours) on both datasets, outperforming several state-of-the-art baselines, including FNN, Stacked GRU, Seq2Seq LSTM, DCRNN, STGCN, and GAMCN.

  • METR-LA Dataset: TrafFormer achieved the best overall performance across all metrics (RMSE, MAE, MAPE) and time steps, except for a very close second place in 6-hour MAPE. It also showed better resilience to increasing prediction horizons compared to most baselines.
  • PEMS-BAY Dataset: While GAMCN performed best on this dataset, TrafFormer was highly competitive, often outperforming DCRNN and other models.
  • Qualitative Analysis: Figures show TrafFormer's ability to better identify the start of rush hours and its relative resilience to sudden traffic drops compared to DCRNN.

Limitations and Future Work

  • Data Quality: The METR-LA dataset's significant amount of missing data presented a challenge, impacting performance compared to the cleaner PEMS-BAY dataset.
  • Model Complexity: While TrafFormer shows promise, the paper doesn't extensively discuss computational complexity or scalability for extremely large-scale deployments.
  • Future Work: The authors suggest that their TrafFormer model provides a strong foundation for future research in long-term traffic prediction. Further exploration of different embedding strategies, attention mechanisms, and incorporating additional external factors (like weather or events) could further enhance prediction accuracy. The paper also highlights the importance of spatial correlation, suggesting that future work could benefit from more sophisticated graph-based approaches integrated with Transformers.

Summary updated

Abstract

Traffic prediction is a flourishing research field due to its importance in human mobility in the urban space. Despite this, existing studies only focus on short-term prediction of up to few hours in advance, with most being up to one hour only. Long-term traffic prediction can enable more comprehensive, informed, and proactive measures against traffic congestion and is therefore an important task to explore. In this paper, we explore the task of long-term traffic prediction; where we predict traffic up to 24 hours in advance. We note the weaknesses of existing models--which are based on recurrent structures--for long-term traffic prediction and propose a modified Transformer model "TrafFormer". Experiments comparing our model with existing hybrid neural network models show the superiority of our model.
BibTeX
@misc{tedjopurnomo2023trafformer,
  author = {David Alexander Tedjopurnomo and Farhana Murtaza Choudhury and A. Kai Qin},
  title = {{TrafFormer: A Transformer Model for Predicting Long-term Traffic}},
  year = {2023},
  eprint = {2302.12388},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.12388},
}
Figure