Strada-LLM: Graph LLM for traffic prediction

Seyed Mohamad Moghadas, Yangxintong Lyu, Bruno Cornelis, Alexandre Alahi, Adrian Munteanu

Abstract

Traffic prediction is a vital component of intelligent transportation systems. By reasoning about traffic patterns in both the spatial and temporal dimensions, accurate and interpretable predictions can be provided. A considerable challenge in traffic prediction lies in handling the diverse data distributions caused by vastly different traffic conditions occurring at different locations. LLMs have been a dominant solution due to their remarkable capacity to adapt to new datasets with very few labeled data samples, i.e., few-shot adaptability. However, existing forecasting techniques mainly focus on extracting local graph information and forming a text-like prompt, leaving LLM-based traffic prediction an open problem. This work presents a probabilistic LLM for traffic forecasting with three highlights. We propose a graph-aware LLM for traffic prediction that considers proximal traffic information. Specifically, by considering the traffic of neighboring nodes as covariates, our model outperforms the corresponding time-series LLM. Furthermore, we adopt a lightweight approach for efficient domain adaptation when facing new data distributions in few-shot fashion. The comparative experiment demonstrates the proposed method outperforms the state-of-the-art LLM-based methods and the traditional GNN-based supervised approaches. Furthermore, Strada-LLM can be easily adapted to different LLM backbones without a noticeable performance drop.

Introduction

Accurate traffic forecasting is essential for modern urban planning and navigation. However, it is a notoriously difficult task because traffic is not just a time-series problem; it is a spatial-temporal one. Road networks exhibit complex dependencies where upstream blockages cause downstream delays (transfer effects) and downstream bottlenecks create upstream backups (feedback effects).

While Large Language Models (LLMs) have shown promise in time-series forecasting, their application to traffic has been limited by "prompt-tuning" paradigms. These methods convert numerical data and graph structures into text descriptions, leading to significant information loss, context window limitations, and computational inefficiency. Strada-LLM addresses these flaws by integrating graph topology directly into the model's latent space.

Methodology: Graph-Aware Architecture

Strada-LLM moves away from text-based prompting in favor of a multivariate probabilistic forecasting framework. The architecture is built on four pillars:

  1. Hierarchical Feature Extractor (HFE): To capture both local and global spatial context, the model uses a $k$-hop sub-graph extractor for immediate neighbors and Laplacian embeddings to understand the global topology of the road network.
  2. LLM-based Backbone: The model utilizes a decoder-only architecture (e.g., Mistral or Llama 2) optimized with Flash-Attention-2. Unlike previous models, it treats traffic data as continuous tokens, preserving numerical precision.
  3. Probabilistic Head: Instead of point-based forecasting, Strada-LLM predicts a Student’s t-distribution. This allows the model to quantify uncertainty, which is critical for managing volatile traffic scenarios.
  4. Efficient Adaptation: To handle new cities or data distributions, the model employs Low-Rank Adaptation (LoRA) and a novel distribution-driven alignment loss based on KL divergence.
Comparison of the Strada-LLM architecture (a) against the prompt-based UniST model (b), highlighting the use of LoRA and direct spatio-temporal embeddings.

Experimental Results

The model was rigorously evaluated against eight real-world datasets, including California highway data (PeMS) and international datasets from Brussels and Nanjing.

Performance Gains

Strada-LLM demonstrated a 17% improvement in Root Mean Square Error (RMSE) for long-term forecasting (up to 150 minutes) compared to state-of-the-art baselines like UniST and FlashST. As shown in the radar chart below, the model consistently achieves lower error rates across zero-shot, few-shot, and long-term scenarios.

Radar chart evaluating RMSE across four key forecasting scenarios; Strada-LLM (Ours) consistently occupies the innermost (lowest error) area.

Efficiency and Robustness

By avoiding text-based prompting, Strada-LLM is 16% more efficient during its adaptation phase. It maintains a lower trainable parameter ratio (1.32%) than its competitors. Furthermore, the model proved to be "backbone-agnostic," showing high performance regardless of whether it was built on Mistral or Llama 2, confirming the robustness of the underlying graph-aware architecture.

Conclusion

Strada-LLM represents a paradigm shift in LLM-based traffic forecasting. By replacing clunky text prompts with direct graph embeddings and probabilistic heads, it achieves superior accuracy and efficiency. Future work will focus on further reducing the computational weight for real-time edge deployment and exploring multi-modal distribution heads to capture even more complex traffic behaviors.

Summary updated

Abstract

Traffic prediction is a vital component of intelligent transportation systems. By reasoning about traffic patterns in both the spatial and temporal dimensions, accurate and interpretable predictions can be provided. A considerable challenge in traffic prediction lies in handling the diverse data distributions caused by vastly different traffic conditions occurring at different locations. LLMs have been a dominant solution due to their remarkable capacity to adapt to new datasets with very few labeled data samples, i.e., few-shot adaptability. However, existing forecasting techniques mainly focus on extracting local graph information and forming a text-like prompt, leaving LLM- based traffic prediction an open problem. This work presents a probabilistic LLM for traffic forecasting with three highlights. We propose a graph-aware LLM for traffic prediction that considers proximal traffic information. Specifically, by considering the traffic of neighboring nodes as covariates, our model outperforms the corresponding time-series LLM. Furthermore, we adopt a lightweight approach for efficient domain adaptation when facing new data distributions in few-shot fashion. The comparative experiment demonstrates the proposed method outperforms the state-of-the-art LLM-based methods and the traditional GNN- based supervised approaches. Furthermore, Strada-LLM can be easily adapted to different LLM backbones without a noticeable performance drop.
BibTeX
@misc{moghadas2024strada,
  author = {Seyed Mohamad Moghadas and Yangxintong Lyu and Bruno Cornelis and Alexandre Alahi and Adrian Munteanu},
  title = {{Strada-LLM: Graph LLM for traffic prediction}},
  year = {2024},
  eprint = {2410.20856},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.20856},
}
Figure