Classical Feature Embeddings Help in BERT-Based Human Mobility Prediction

Yunzhi Liu, Haokai Tan, Rushi Kanjaria, Lihuan Li, Flora D. Salim

TL;DR

This paper introduces STaBERT, a novel BERT-based model for human mobility prediction that significantly improves accuracy by incorporating Point of Interest (POI) and temporal features. By enriching location embeddings with semantic and temporal context, STaBERT achieves state-of-the-art results on both single-city and multi-city prediction tasks, outperforming previous methods by a substantial margin.

What Problem Does This Paper Address?

Predicting human movement patterns is crucial for urban planning, disaster response, and public health. Existing models often struggle to capture the full richness of mobility data. Many focus solely on location sequences or treat temporal information as a secondary input. A significant gap exists in effectively leveraging the semantic context provided by Points of Interest (POIs) – places like shops, parks, or offices – which strongly influence human behavior. While some attempts have been made to incorporate POI data, they often lack the sophisticated modeling capabilities of modern deep learning architectures like Transformers. This paper aims to bridge this gap by developing a model that seamlessly integrates both semantic (POI) and temporal information into a unified representation for more accurate human mobility prediction.

Methods

The core of this work is the STaBERT (Semantic-Temporal aware BERT) model. It builds upon the LP-BERT architecture, a winning solution from the 2023 HuMob Challenge, which treats trajectory prediction as a BERT-based interpolation task. The key innovations are:

  • POI Embeddings: POI data, represented by categories and visit counts, is transformed into dense vector embeddings. A dedicated module uses two embedding layers and a mean pooling operation to create a fixed-length, location-level representation that captures the semantic meaning of POIs at a given location.
  • Temporal Descriptors: Beyond basic time information, STaBERT incorporates derived temporal features such as the day of the week, an indicator for weekends, the activity period (e.g., "active," "rest"), and the time of day (e.g., AM/PM). These are also embedded and integrated into the model's input.
  • Unified Representation: Both POI and temporal embeddings are combined with existing location and time features, then fed into a Transformer encoder. This allows the model to learn a comprehensive, semantically enriched representation of human mobility patterns.

Datasets

The experiments were conducted using data from the HuMob Challenge 2023 and 2024.

  • HuMob Challenge 2023: This dataset focuses on a dense Japanese metropolitan area, providing movement data for 100,000 individuals over 90 days at 30-minute intervals. Tasks involved predicting future movements for subsets of individuals. Optional POI data (85-dimensional vectors per cell) was also provided.
  • HuMob Challenge 2024: This dataset extends the problem to multi-city prediction, using data from cities B, C, and D, with city A serving as a source of extensive movement data.

Key Insights and Ideas

  • Semantics Matter: The inclusion of POI information significantly boosts prediction accuracy, demonstrating that understanding where people go and what is there is as important as when they move.
  • Temporal Nuances are Key: Beyond just timestamps, derived temporal features like day of the week and activity periods provide crucial context for predicting daily routines and deviations.
  • Unified Modeling is Powerful: Combining semantic and temporal features within a single Transformer-based architecture allows for a more holistic understanding of mobility patterns, leading to superior performance.
  • Robustness in Scenarios: While POI data's impact is most pronounced in normal scenarios, its contribution in emergency situations is less clear, potentially due to outdated POI information in such contexts.

Key Results

STaBERT achieved substantial improvements in prediction accuracy:

  • Single-City Prediction: The GEO-BLEU score increased from a baseline of 0.34 to 0.75.
  • Multi-City Prediction: The GEO-BLEU score rose from 0.34 to 0.56.
  • Contribution of Features: Experiments showed that adding POI data alone significantly improved GEO-BLEU scores, and further adding temporal descriptors led to even greater refinement, approaching near-perfect levels in some cases (e.g., user 80004 reaching 0.948 GEO-BLEU with both features).
  • Efficiency: The model maintains a good balance between accuracy and computational efficiency, with added temporal features not significantly increasing training time.

Limitations and Future Work

  • POI Data Availability in Emergencies: The paper notes that in emergency scenarios, the effectiveness of POI data might be limited if the data is not updated to reflect the actual situation (e.g., closures). This suggests a need for dynamic POI data or alternative strategies in such contexts.
  • Future Directions: The authors suggest that the embedding module is flexible and can be extended to incorporate other spatial descriptors, such as land-use information or clustering-based location types, to further enhance prediction capabilities, especially in scenarios where POI data might be scarce or unreliable.

Summary updated

Abstract

Human mobility forecasting is crucial for disaster relief, city planning, and public health. However, existing models either only model location sequences or include time information merely as auxiliary input, thereby failing to leverage the rich semantic context provided by points of interest (POIs). To address this, we enrich a BERT-based mobility model with derived temporal descriptors and POI embeddings to better capture the semantics underlying human movement. We propose STaBERT (Semantic-Temporal aware BERT), which integrates both POI and temporal information at each location to construct a unified, semantically enriched representation of mobility. Experimental results show that STaBERT significantly improves prediction accuracy: for single-city prediction, the GEO-BLEU score improved from 0.34 to 0.75; for multi-city prediction, from 0.34 to 0.56.
BibTeX
@inproceedings{liu2025classical,
  author = {Yunzhi Liu and Haokai Tan and Rushi Kanjaria and Lihuan Li and Flora D. Salim},
  title = {{Classical Feature Embeddings Help in BERT-Based Human Mobility Prediction}},
  booktitle = {SIGSPATIAL/GIS},
  year = {2025},
  url = {https://openreview.net/forum?id=cMggx7mO3w},
}
Figure