Introduction
Predicting long-term human mobility is a cornerstone of modern urban planning, disaster management, and public health initiatives. However, cross-city prediction—applying models trained in one urban environment to another—presents significant challenges. These include data sparsity in smaller cities, the inherent behavioral complexity of human movement, and the high degree of heterogeneity between different metropolitan infrastructures. To address these issues, we present ST-MoE-BERT, a framework designed to leverage knowledge from data-rich environments to improve predictions in data-scarce ones.
Methodology
The ST-MoE-BERT architecture integrates the sequence-modeling capabilities of BERT with a Mixture-of-Experts (MoE) layer to capture complex spatial-temporal dependencies.
Architecture Components
- BERT Backbone: The model utilizes a BERT-based architecture to process temporal sequences of locations. By treating spatial trajectories as "sentences" of grid cells, the self-attention mechanism identifies critical historical nodes relevant to future movement.
- Mixture-of-Experts (MoE) Layer: Replacing standard feed-forward networks, the MoE layer contains eight expert networks. A routing mechanism selects the top two experts for each input, allowing the model to develop specialized knowledge for different mobility patterns.
- Differential Transfer Learning: To facilitate cross-city adaptation, the model employs a unique transfer strategy. During fine-tuning on a target city, the learning rate for location embeddings is set ten times higher than for the rest of the network. This allows the model to rapidly re-map its spatial understanding to a new city's grid without losing the generalized mobility logic learned during pre-training.
Datasets and Experimental Setup
The framework was evaluated using mobility data from the HuMob’24 challenge, encompassing four metropolitan areas (Cities A, B, C, and D). Each city is represented as a 200x200 grid with 500m resolution.
City A served as the primary source for pre-training, containing 100,000 users. Cities B, C, and D, with significantly smaller user bases (25k, 20k, and 6k respectively), served as target cities to test the model's effectiveness in data-scarce scenarios. The data spans 75 days with 30-minute sampling intervals.
Results and Discussion
ST-MoE-BERT demonstrates substantial improvements over baseline methods, including Historical Frequency (HF) and standard BERT models. Evaluation metrics included GEO-BLEU for precision, Dynamic Time Warping (DTW) for sequence similarity, and standard Accuracy.
The model achieved an average performance boost of:
- 8.29% in GEO-BLEU
- 9.90% in DTW
- 10.76% in Accuracy
Ablation studies highlighted the critical role of pre-training on City A. The MoE layer proved essential for managing city heterogeneity, while the differential learning rate strategy enabled efficient spatial re-mapping.
Conclusion
ST-MoE-BERT offers a robust solution for cross-city mobility prediction by combining the strengths of transformer-based sequence modeling with the flexibility of Mixture-of-Experts. Future work will explore the integration of Large Language Models (LLMs) to incorporate external contextual data, such as weather and social trends, and investigate methods for handling finer-grained spatial resolutions.
Summary updated