Forecaster: A Graph Transformer for Forecasting Spatial and Time-Dependent Data

Yang Li, José M. F. Moura

Abstract

Spatial and time-dependent data forecasting is critical for urban planning and resource management but remains challenging due to complex spatial dependencies, long-range temporal patterns, and data non-stationarity. We present Forecaster, a graph Transformer architecture that addresses these issues by learning a parsimonious dependency graph via Gaussian Markov Random Fields (GMRF). This graph is then used to sparsify the Transformer's internal layers, effectively regularizing the model and focusing its attention on relevant spatial neighbors. Evaluated on a massive NYC taxi demand dataset, Forecaster significantly outperforms state-of-the-art baselines, demonstrating the power of data-driven topology in deep learning architectures.

Introduction

Predicting variables that evolve across both space and time—such as regional economic indicators, traffic flow, or ride-hailing demand—is a cornerstone of modern data science. However, standard models often fail to capture the nuances of these systems. Proximity in physical space does not always imply a strong functional correlation; for instance, two distant transit hubs may be more synchronized than a hub and an adjacent residential street. Furthermore, temporal patterns can span weeks, and sudden events like storms can disrupt stationarity.

Methodology: The Forecaster Architecture

The Forecaster model employs a two-stage pipeline to bridge the gap between structural graph learning and sequence modeling.

1. Learning the Dependency Graph

Rather than relying on predefined distance metrics like Euclidean distance, Forecaster learns the underlying spatial structure directly from historical data. Using Gaussian Markov Random Fields (GMRF) and the graphical lasso algorithm, the model estimates a precision matrix that represents the conditional dependency between locations. This process prunes insignificant connections, resulting in a sparse graph where an edge exists only if two locations are truly interdependent.

2. The Sparse Graph Transformer

The core innovation lies in the integration of this learned graph into a Transformer architecture. In a standard Transformer, linear layers are fully connected, which can lead to overfitting on noise from unrelated spatial locations. Forecaster introduces Sparse Linear Layers, where connections are restricted based on the learned dependency graph.

The Forecaster architecture, featuring sparse multi-head attention and feed-forward layers within an encoder-decoder framework.

As shown in the architecture above, this sparsification is applied to the Multi-Head Attention and Feedforward layers. Each neuron corresponds to a spatial location, and its receptive field is limited to its neighbors in the dependency graph. This "hard-coding" of spatial relationships allows the attention mechanism to focus on temporal dynamics and non-stationary shifts without being distracted by irrelevant spatial noise.

Experimental Evaluation

The model was tested on the NYC Taxi Dataset, encompassing 7.5 years of demand data across 996 high-traffic locations in Manhattan. This was supplemented with hourly weather data and calendar features to account for heterogeneity.

Key Insights: Data-Driven Topology

A significant finding was the extreme sparsity of the learned dependencies. On average, a location in Manhattan was found to depend on only ~2.5 other locations.

A visualization of the learned dependency structure, showing a highly sparse precision matrix where most location pairs are conditionally independent.

The figure above illustrates this sparsity. By leveraging this structure, Forecaster acts as a powerful regularizer. It prevents the model from learning spurious correlations between distant, unrelated neighborhoods, leading to more robust predictions.

Results and Discussion

Forecaster demonstrated superior performance across all metrics when compared to traditional and deep learning baselines:

  • Accuracy Gains: It achieved an 8.82% reduction in RMSE and a 9.62% reduction in MAPE compared to a standard "dense" Transformer.
  • Comparison to SOTA: Against DCRNN (a graph-based RNN), Forecaster improved MAPE by 19.41%, highlighting the advantage of the Transformer's attention mechanism for long-range temporal dependencies.
  • Spatial Awareness: Qualitative analysis confirmed that the model captured functional dependencies, such as the link between major transit hubs like Grand Central and Penn Station, which are physically distant but operationally linked.

Conclusion

Forecaster represents a significant step forward in spatial-temporal forecasting by replacing "ad-hoc" spatial assumptions with a data-driven dependency graph. While the current model uses a static graph, future work could explore dynamic graph learning to adapt to evolving urban landscapes. Nonetheless, the success of Forecaster in the NYC taxi domain suggests broad applicability to other complex, multi-variate time-series problems.

Summary updated

Abstract

Spatial and time-dependent data is of interest in many applications. This task is difficult due to its complex spatial dependency, long-range temporal dependency, data non-stationarity, and data heterogeneity. To address these challenges, we propose Forecaster, a graph Transformer architecture. Specifically, we start by learning the structure of the graph that parsimoniously represents the spatial dependency between the data at different locations. Based on the topology of the graph, we sparsify the Transformer to account for the strength of spatial dependency, long-range temporal dependency, data non-stationarity, and data heterogeneity. We evaluate Forecaster in the problem of forecasting taxi ride-hailing demand and show that our proposed architecture significantly outperforms the state-of-the-art baselines.
BibTeX
@misc{li2019forecaster,
  author = {Yang Li and José M. F. Moura},
  title = {{Forecaster: A Graph Transformer for Forecasting Spatial and Time-Dependent Data}},
  year = {2019},
  eprint = {1909.04019},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.04019},
}
Figure