Abstract
Spatial and time-dependent data forecasting is critical for urban planning and resource management but remains challenging due to complex spatial dependencies, long-range temporal patterns, and data non-stationarity. We present Forecaster, a graph Transformer architecture that addresses these issues by learning a parsimonious dependency graph via Gaussian Markov Random Fields (GMRF). This graph is then used to sparsify the Transformer's internal layers, effectively regularizing the model and focusing its attention on relevant spatial neighbors. Evaluated on a massive NYC taxi demand dataset, Forecaster significantly outperforms state-of-the-art baselines, demonstrating the power of data-driven topology in deep learning architectures.
Introduction
Predicting variables that evolve across both space and time—such as regional economic indicators, traffic flow, or ride-hailing demand—is a cornerstone of modern data science. However, standard models often fail to capture the nuances of these systems. Proximity in physical space does not always imply a strong functional correlation; for instance, two distant transit hubs may be more synchronized than a hub and an adjacent residential street. Furthermore, temporal patterns can span weeks, and sudden events like storms can disrupt stationarity.
Methodology: The Forecaster Architecture
The Forecaster model employs a two-stage pipeline to bridge the gap between structural graph learning and sequence modeling.
1. Learning the Dependency Graph
Rather than relying on predefined distance metrics like Euclidean distance, Forecaster learns the underlying spatial structure directly from historical data. Using Gaussian Markov Random Fields (GMRF) and the graphical lasso algorithm, the model estimates a precision matrix that represents the conditional dependency between locations. This process prunes insignificant connections, resulting in a sparse graph where an edge exists only if two locations are truly interdependent.
2. The Sparse Graph Transformer
The core innovation lies in the integration of this learned graph into a Transformer architecture. In a standard Transformer, linear layers are fully connected, which can lead to overfitting on noise from unrelated spatial locations. Forecaster introduces Sparse Linear Layers, where connections are restricted based on the learned dependency graph.
As shown in the architecture above, this sparsification is applied to the Multi-Head Attention and Feedforward layers. Each neuron corresponds to a spatial location, and its receptive field is limited to its neighbors in the dependency graph. This "hard-coding" of spatial relationships allows the attention mechanism to focus on temporal dynamics and non-stationary shifts without being distracted by irrelevant spatial noise.
Experimental Evaluation
The model was tested on the NYC Taxi Dataset, encompassing 7.5 years of demand data across 996 high-traffic locations in Manhattan. This was supplemented with hourly weather data and calendar features to account for heterogeneity.
Key Insights: Data-Driven Topology
A significant finding was the extreme sparsity of the learned dependencies. On average, a location in Manhattan was found to depend on only ~2.5 other locations.
The figure above illustrates this sparsity. By leveraging this structure, Forecaster acts as a powerful regularizer. It prevents the model from learning spurious correlations between distant, unrelated neighborhoods, leading to more robust predictions.
Results and Discussion
Forecaster demonstrated superior performance across all metrics when compared to traditional and deep learning baselines:
- Accuracy Gains: It achieved an 8.82% reduction in RMSE and a 9.62% reduction in MAPE compared to a standard "dense" Transformer.
- Comparison to SOTA: Against DCRNN (a graph-based RNN), Forecaster improved MAPE by 19.41%, highlighting the advantage of the Transformer's attention mechanism for long-range temporal dependencies.
- Spatial Awareness: Qualitative analysis confirmed that the model captured functional dependencies, such as the link between major transit hubs like Grand Central and Penn Station, which are physically distant but operationally linked.
Conclusion
Forecaster represents a significant step forward in spatial-temporal forecasting by replacing "ad-hoc" spatial assumptions with a data-driven dependency graph. While the current model uses a static graph, future work could explore dynamic graph learning to adapt to evolving urban landscapes. Nonetheless, the success of Forecaster in the NYC taxi domain suggests broad applicability to other complex, multi-variate time-series problems.
Summary updated