Abstract
Spatio-temporal forecasting is pivotal in numerous real-world applications, including transportation planning, energy management, and climate monitoring. In this work, we aim to harness the reasoning and generalization abilities of Pre-trained Language Models (PLMs) for more effective spatio-temporal forecasting, particularly in data-scarce scenarios. However, recent studies uncover that PLMs, which are primarily trained on textual data, often falter when tasked with modeling the intricate correlations in numerical time series, thereby limiting their effectiveness in comprehending spatio-temporal data. To bridge the gap, we propose RePST, a semantic-oriented PLM reprogramming framework tailored for spatio-temporal forecasting. Specifically, we first propose a semantic-oriented decomposer that adaptively disentangles spatially correlated time series into interpretable sub-components, which facilitates PLM to understand sophisticated spatio-temporal dynamics via a divide-and-conquer strategy. Moreover, we propose a selective discrete reprogramming scheme, which introduces an expanded spatio-temporal vocabulary space to project spatio-temporal series into discrete representations. This scheme minimizes the information loss during reprogramming and enriches the representations derived by PLMs. Extensive experiments on real-world datasets show that the proposed RePST outperforms twelve state-of-the-art baseline methods, particularly in data-scarce scenarios, highlighting the effectiveness and superior generalization capabilities of PLMs for spatio-temporal forecasting.
Introduction
Spatio-temporal (ST) forecasting—predicting future states in systems like traffic networks, power grids, or weather patterns—is notoriously difficult. These systems involve intricate dependencies across both time and space. While deep learning has made strides here, most models are "one-task-one-model," requiring massive amounts of data to learn from scratch.
Recently, researchers have tried to use Pre-trained Language Models (PLMs) for this task, hoping to leverage their inherent reasoning and few-shot generalization abilities. However, there is a major "modality gap": PLMs are built for the semantic structure of text, not the raw, noisy, and highly coupled numerical streams of ST data. Simply feeding numbers into a Transformer doesn't work well because the model fails to "understand" the underlying physical dynamics. RePST (Semantic-Oriented Reprogramming) is designed to bridge this gap.
Methodology
The RePST framework introduces two main innovations to align numerical spatio-temporal data with the semantic processing capabilities of PLMs:
- Semantic-Oriented Spatio-Temporal Decomposer: Instead of feeding raw data, RePST uses Koopman Theory (specifically Dynamic Mode Decomposition) to disentangle the signal. It breaks complex ST dynamics into interpretable "modes"—separating dominant physical patterns (like periodic traffic cycles) from random noise. This "divide-and-conquer" strategy makes the data much more digestible for a language model.
- Selective Discrete Reprogramming: To turn these numerical modes into something a PLM can process, the authors propose a selective vocabulary scheme. Using a differentiable mask (via the Gumbel-Softmax trick), the model learns to select the most relevant "word tokens" from the PLM’s existing vocabulary to represent specific ST patterns. This creates a bridge where numerical patches are represented as high-dimensional "textual" embeddings that the frozen PLM backbone (like GPT-2) can then process using its pre-trained attention layers.
The entire backbone of the PLM remains frozen, meaning the model relies on its pre-existing knowledge of sequences and relationships rather than being fine-tuned on the numbers themselves.
Experimental Setup
The model was evaluated on six diverse, real-world benchmarks across three domains:
- Traffic: METR-LA and PEMS-BAY (highway sensors), Beijing Taxi (inflow/outflow), and NYC Bike (station demand).
- Energy: Solar Energy (output from 137 PV plants).
- Environment: Air Quality (six pollutants across 35 stations).
Key Results and Insights
RePST outperformed 12 state-of-the-art baselines, including specialized Graph Neural Networks (GNNs) and other PLM-based time-series models.
- Decomposition is the Key to "Understanding": The authors found that their Koopman-based approach is superior because it extracts modes that align with actual physical phenomena, making them easier for the PLM to interpret.
- Few-Shot Excellence: In scenarios with only one day of training data, RePST consistently beat models trained on much larger datasets. For example, on the METR-LA traffic dataset, it achieved a Mean Absolute Error (MAE) of 5.63, significantly lower than the next best baseline (6.35).
- Zero-Shot Prowess: The model demonstrated strong performance in cross-domain transfers (e.g., training on NYC traffic and testing on Beijing taxi data), proving that it learns universal spatio-temporal patterns rather than just memorizing a specific dataset.
- Stability: Unlike many models whose accuracy plummets as the prediction horizon increases, RePST remained relatively stable even when predicting 48 time steps into the future.
Conclusion
RePST demonstrates that by "translating" physics into a language the PLM understands, we can effectively leverage pre-trained reasoning for numerical forecasting. While computational complexity and the exact theoretical nature of this cross-modality reasoning remain areas for future study, RePST sets a new benchmark for data-efficient and generalizable spatio-temporal forecasting.
Summary updated