TL;DR
This paper introduces STD-PLM, a novel framework that leverages Pre-trained Language Models (PLMs) for both spatial-temporal forecasting and imputation. By developing specialized tokenizers and a Sandglass Attention module, STD-PLM effectively captures complex spatial-temporal dependencies, achieving state-of-the-art performance and demonstrating strong few-shot and zero-shot learning capabilities.
What Problem Does This Paper Address?
Real-world intelligent systems often rely on predicting future states (forecasting) or filling in missing information (imputation) for spatial-temporal data, such as traffic flow or sensor readings. Existing methods typically excel at one task but not both, and they struggle with scenarios where limited training data is available (few-shot/zero-shot learning). While PLMs have shown great promise in various domains, adapting them to the unique characteristics of spatial-temporal data—like complex correlations and graph structures—has been challenging. This paper aims to bridge this gap by creating a unified, efficient PLM-based framework that handles both forecasting and imputation, while also excelling in low-data regimes.
Methods
The core of STD-PLM lies in its innovative approach to preparing spatial-temporal data for a PLM:
- Spatial-Temporal Embeddings: The model first creates embeddings that capture both the static characteristics of nodes (topology-aware node embeddings using graph Laplacian eigenvectors) and the dynamic temporal patterns (periodic-aware time embeddings for time-of-day and day-of-week).
- Spatial-Temporal Tokenizers: These modules convert the embedded data into sequences of tokens that a PLM can process.
- Spatial Tokenizer: Aggregates temporal information for each node to create node-level spatial tokens, enabling the PLM to understand spatial correlations.
- Temporal Tokenizer: Encapsulates the overall system state and trends across all nodes at each time step, generating state and trend tokens.
- Sandglass Attention (SGA) Module: To address the computational burden of processing a large number of spatial tokens, SGA first uses a "precoder" to aggregate node-level spatial tokens into fewer region-level tokens. This allows the PLM to capture non-pairwise and higher-order spatial-temporal correlations more efficiently. A "decoder" then restores the original token length.
- Constrained Loss Function: A specialized loss function is introduced to guide the SGA module, ensuring that learned region-level spatial tokens reflect the original graph structure and preventing overfitting by encouraging a more even distribution of attention weights.
- PLM Fine-tuning: The PLM is fine-tuned using partial freezing and Low-Rank Adaptation (LoRA) on its attention, position embedding, and layer normalization layers, making the process more efficient.
Datasets
The experiments were conducted on four real-world traffic datasets:
- PEMS03
- PEMS04
- PEMS07
- PEMS08
For imputation tasks, missing data patterns (Random Missing - RM, and Spatial-Temporal Continuous Missing - CM) were generated on the PEMS08 dataset with a 70% missing rate.
Key Insights and Ideas
- PLMs are powerful for spatial-temporal data: The paper demonstrates that with appropriate adaptation (tokenization, embedding), PLMs can effectively learn complex spatial-temporal patterns, outperforming traditional time-series models.
- Unified framework is feasible and effective: STD-PLM shows that a single model can achieve high performance on both forecasting and imputation tasks, simplifying deployment and reducing development costs.
- Efficiency is crucial: The Sandglass Attention module is a key innovation, significantly reducing computational overhead by creating region-level tokens, making PLM-based approaches more practical.
- Few-shot and zero-shot learning are achievable: The model's ability to perform well with very little training data or even transfer to unseen datasets highlights its strong generalization capabilities, crucial for real-world applications where data collection is costly.
- Topology matters: Explicitly incorporating graph structure through topology-aware embeddings is vital for understanding spatial relationships.
Key Results
- Forecasting: STD-PLM achieved the best or second-best results across all four datasets in forecasting tasks, outperforming both conventional deep learning models and other PLM-based baselines. For example, on PEMS08, it achieved an MAE of 13.31, RMSE of 23.19, and MAPE of 8.84%.
- Imputation: The model demonstrated state-of-the-art performance in imputation tasks, showing substantial improvement over baselines, particularly in the RM 70% case on PEMS08.
- Few-Shot Learning: With only 5% of the training data, STD-PLM achieved performance comparable to LSTM trained on the full dataset. At 20% of training data, it surpassed ASTGCN.
- Zero-Shot Learning: The trained model showed acceptable performance when directly transferred to datasets with different temporal scopes and graph structures without further training.
- Efficiency: STD-PLM boasts the fastest inference speed among the compared models, despite having a large number of parameters, thanks to the SGA module and efficient PLM fine-tuning.
Limitations and Future Work
- Computational Limitations: The authors acknowledge that due to computational constraints, they could not test larger PLMs (e.g., 7B, 30B parameters), which might offer even better performance.
- Emergent Phenomena: The paper did not explore whether a unified pre-trained model on a large spatial-temporal dataset could exhibit emergent phenomena similar to those seen in large language models.
- Future Work: Future research could focus on leveraging larger PLMs and exploring the potential for emergent capabilities in spatial-temporal data understanding. The authors also suggest that adequate computational power and data would be beneficial for further exploration.
Summary updated