Exploring Transformer and Graph Convolutional Networks for Human Mobility Modeling

Riccardo Corrias, Martin Gjoreski, Marc Langheinrich

TL;DR

This paper explores the application of cutting-edge deep learning models, Transformers (specifically GPT-like architectures) and Graph Convolutional Networks (GCNs), to the problem of predicting an individual's next location. The study finds that these advanced models perform comparably to existing state-of-the-art methods, suggesting a promising future for their use in more complex mobility modeling tasks.

What Problem Does This Paper Address?

Understanding and predicting human mobility patterns is crucial for various societal applications, from urban planning and traffic management to public health initiatives like tracking disease spread. A key aspect of this is "next-place prediction," which aims to forecast where an individual will go next based on their past movements. While traditional methods like Markov models and early machine learning techniques have been used, they often struggle with the complexity and scale of real-world mobility data. The authors identify a gap in leveraging recent AI advancements, specifically Transformers and GCNs, which have shown remarkable success in other domains like natural language processing and image analysis, for this mobility prediction task.

Methods

The researchers adapted two general-purpose deep learning architectures for next-place prediction:

  • Transformer-based model (TFT): Inspired by the Temporal Fusion Transformer (TFT), this model uses a sequence-to-sequence architecture with attention mechanisms to capture long-term dependencies in time-series data. It processes historical location data, static user information, and known future inputs (like time of day) to predict the next location.
  • Graph Convolutional Network (GCN) model (MTGNN): Based on the Multivariate Time Series Graph Neural Network (MTGNN), this approach frames the problem as a graph where each visited Point of Interest (POI) is a node. It uses graph learning to dynamically infer relationships between POIs and temporal convolution to capture sequential patterns, aiming to model hidden interdependencies between locations.

As a baseline, they also implemented and evaluated the Flashback-LSTM model, a state-of-the-art method specifically designed for sparse mobility data.

Datasets

The study utilized three distinct datasets to evaluate the models:

  • Foursquare: A sparse dataset derived from user check-ins, containing millions of check-ins across numerous locations globally.
  • Gowalla: Another sparse, check-in-based dataset, similar to Foursquare but with a smaller number of check-ins.
  • Breadcrumbs: A dense dataset collected via continuous GPS sensing, offering a higher frequency of location data compared to the check-in datasets.

To further investigate model behavior, they also created several sub-datasets from Foursquare and Gowalla based on user demographics (country, city, gender) and social network information (Twitter followers, number of friends).

Key Insights and Ideas

  • Transformers and GCNs are viable for mobility: The study demonstrates that advanced architectures like Transformers and GCNs, which have excelled in other AI domains, can be effectively adapted and applied to the complex task of human mobility prediction.
  • Data sparsity is a major challenge: The performance of all models was significantly impacted by data sparsity. Sparse, check-in-based datasets yielded lower accuracy compared to dense, continuous GPS data.
  • Specialized models still hold an edge on sparse data: For sparse datasets, the Flashback-LSTM, specifically designed for this challenge, slightly outperformed the more general Transformer and GCN models. This suggests that incorporating mechanisms like "flashbacks" to leverage historical hidden states can be beneficial.
  • Dense data levels the playing field: On the dense Breadcrumbs dataset, all three models performed similarly, indicating that as data becomes richer and more continuous (like from smartphones), the advantage of specialized sparse-data models diminishes.
  • Future potential for GPT/GCN: Despite not significantly surpassing the current state-of-the-art on sparse data, the comparable performance of GPT and GCN models suggests they have substantial potential to lead future advancements, especially as dense mobility data becomes more prevalent.

Key Results

  • Overall Performance: On the sparse Foursquare and Gowalla datasets, Flashback-LSTM achieved slightly higher ACC@1 scores (e.g., 0.2678 for Foursquare) compared to TFT (0.2577) and MTGCN (0.2356). The differences were generally within 3.5 percentage points.
  • Dense Data Performance: On the dense Breadcrumbs dataset, all three models achieved similar high accuracy scores (around 0.94 ACC@1).
  • Sub-dataset Analysis: Experiments on Foursquare sub-datasets showed that model performance varied based on user characteristics, but the general trend of Flashback-LSTM slightly outperforming TFT and MTGCN on sparse data held. TFT slightly outperformed MTGCN across most sub-datasets.
  • Training Stability: Learning curves for all models indicated stable training processes with no significant overtraining.

Limitations and Future Work

  • Hyperparameter Optimization: The study did not perform extensive hyperparameter tuning for the GPT and GCN models, which could potentially improve their performance. Bayesian optimization is suggested as a future direction.
  • Data Privacy: The current centralized approach raises privacy concerns. The authors suggest exploring federated learning and cohort-based modeling as privacy-preserving alternatives.
  • Data Bias and Completeness: Models trained on potentially biased or incomplete datasets may exhibit reduced accuracy for certain demographic groups or regions.
  • Inclusion of Socio-economic Factors: Future work could explore incorporating factors like economic situation and educational background, which are currently not well-addressed by these models.
  • Interpretability: The complexity of GPT and GCN models makes them difficult to interpret. Developing Explainable AI (XAI) techniques is crucial for policymakers and stakeholders.
  • Dynamic Mobility: Models need to be updated frequently to capture the dynamic nature of human mobility.
  • Fairness: Avoiding discriminatory outcomes is another important consideration for future research.

Summary updated

Abstract

The estimation of human mobility patterns is essential for many components of developed societies, including the planning and management of urbanization, pollution, and disease spread. One important type of mobility estimator is the next-place predictors, which use previous mobility observations to anticipate an individual’s subsequent location. So far, such predictors have not yet made use of the latest advancements in artificial intelligence methods, such as General Purpose Transformers (GPT) and Graph Convolutional Networks (GCNs), which have already achieved outstanding results in image analysis and natural language processing. This study explores the use of GPT- and GCN-based models for next-place prediction. We developed the models based on more general time series forecasting architectures and evaluated them using two sparse datasets (based on check-ins) and one dense dataset (based on continuous GPS data). The experiments showed that GPT-based models slightly outperformed the GCN-based models with a difference in accuracy of 1.0 to 3.2 percentage points (p.p.). Furthermore, Flashback-LSTM—a state-of-the-art model specifically designed for next-place prediction on sparse datasets—slightly outperformed the GPT-based and GCN-based models on the sparse datasets (1.0 to 3.5 p.p. difference in accuracy). However, all three approaches performed similarly on the dense dataset. Given that future use cases will likely involve dense datasets provided by GPS-enabled, always-connected devices (e.g., smartphones), the slight advantage of Flashback on the sparse datasets may become increasingly irrelevant. Given that the performance of the relatively unexplored GPT- and GCN-based solutions was on par with state-of-the-art mobility prediction models, we see a significant potential for them to soon surpass today’s state-of-the-art approaches.
BibTeX
@inproceedings{corrias2023exploring,
  author = {Riccardo Corrias and Martin Gjoreski and Marc Langheinrich},
  title = {{Exploring Transformer and Graph Convolutional Networks for Human Mobility Modeling}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=ZhaHjy4NOQ},
}
Figure