Abstract
Spatio-temporal forecasting is a critical task in urban computing, requiring models to simultaneously capture complex temporal dependencies and intricate spatial relationships. While Large Language Models (LLMs) have demonstrated exceptional capabilities in sequence modeling, their inherent 1D architecture often struggles to represent the graph-like spatial structures of urban networks. This paper introduces ST-LINK, a framework that transforms LLMs into spatially-aware forecasters. By integrating a 2D rotary attention mechanism and a dynamic memory-retrieval system, ST-LINK achieves state-of-the-art performance in traffic and demand forecasting while significantly reducing computational overhead compared to existing LLM-based approaches.
Introduction
Predicting urban dynamics—such as traffic flow, congestion, or bike-sharing demand—requires a dual understanding of "when" an event occurs (temporal) and "where" it happens (spatial). Traditional deep learning models often utilize Graph Convolutional Networks (GCNs) to handle spatiality, but these can be difficult to scale and integrate with the sequential reasoning strengths of LLMs.
Previous attempts to adapt LLMs for spatio-temporal tasks have largely relied on prompt engineering or external spatial embeddings. However, these methods often treat spatial information as an auxiliary input rather than a fundamental part of the model's internal reasoning. ST-LINK addresses this gap by modifying the internal architecture of the LLM to make it natively spatially aware, allowing for more robust and efficient forecasting.
Methodology
ST-LINK modifies a standard LLM (specifically GPT-2 in the authors' experiments) through three primary innovations:
1. Spatially-Enhanced Attention (SE-Attention)
Standard LLMs utilize Rotary Positional Embeddings (RoPE) to track the relative order of tokens in a 1D sequence. ST-LINK extends this concept into two dimensions. The SE-Attention mechanism decomposes query and key embeddings into distinct temporal and spatial components. By applying rotational transformations that account for both time steps and the physical distance/connectivity between sensors, the model learns spatial dependencies as a natural part of its attention flow. This allows the LLM to "see" the city's road network directly within its core attention layers.
2. Memory Retrieval Feed-Forward Network (MRFFN)
To improve long-term stability and handle recurring patterns, ST-LINK replaces standard feed-forward layers with a Memory Retrieval Feed-Forward Network. Inspired by Retrieval-Augmented Generation (RAG), this component maintains a dynamic memory bank of historical patterns. When processing current data, the model retrieves similar historical contexts and uses a Mixture-of-Experts (MoE) layer to route the data to specialized sub-networks. This ensures that the model can leverage past "typical" scenarios (e.g., a standard Monday morning rush hour) to stabilize its predictions.
3. Partially Frozen Training
To maintain computational efficiency and leverage the pre-trained knowledge of the LLM, the authors employ a partially frozen training strategy. The lower layers of the LLM remain frozen, while only the upper layers and the newly introduced spatial and memory modules are fine-tuned. This approach preserves the model's general sequence-modeling capabilities while specializing it for spatio-temporal data.
Experimental Setup
The framework was evaluated on four benchmark datasets representing diverse urban forecasting challenges:
- Demand Prediction: NYCTaxi (266 stations) and Citi Bike (250 stations) in New York City.
- Traffic Speed Forecasting: METR-LA (207 sensors in Los Angeles) and PEMS-BAY (325 sensors in the Bay Area).
Performance was measured using standard metrics: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Mean Absolute Percentage Error (MAPE).
Results and Discussion
ST-LINK demonstrated significant improvements over state-of-the-art baselines, including GMAN and ST-LLM.
Predictive Accuracy
The model consistently achieved the lowest error rates across all datasets. In the METR-LA traffic speed forecasting task, ST-LINK outperformed competitors across 15-minute, 30-minute, and 1-hour horizons. Qualitative analysis showed that the model is particularly adept at capturing "sudden events," such as accidents or unexpected congestion, where traditional models often lag or fail to adjust.
Computational Efficiency
One of the most striking results is the efficiency of the ST-LINK architecture. Compared to the previous state-of-the-art LLM-based model (ST-LLM), ST-LINK achieved:
- 87% reduction in parameters (10M vs. 82M).
- 38.2% faster training times.
- 37.5% faster inference speeds.
These gains are attributed to the sparse Mixture-of-Experts approach within the memory module and the efficient 2D attention mechanism, which avoids the need for massive, fully-connected spatial embedding layers.
Conclusion
ST-LINK represents a significant step forward in integrating spatial awareness into Large Language Models. By treating spatial relationships as rotations in a high-dimensional space and utilizing a dynamic memory bank, the framework achieves superior accuracy with a fraction of the parameters required by previous models.
Future research will focus on addressing the risk of "catastrophic forgetting" in the rapidly updating memory module and exploring how these spatial mechanisms scale with even larger foundation models. Additionally, the authors suggest that the ST-LINK framework could be extended to other spatio-temporal domains, such as climate modeling and energy grid management.
Summary updated