TL;DR
This paper introduces STAT, a novel Transformer-based model for Point-of-Interest (POI) recommendations in location-based social networks. STAT addresses limitations of existing graph-based methods by incorporating a geographically aware attention mechanism and a temporally aware sampling strategy to better capture user preferences and handle large-scale data.
What Problem Does This Paper Address?
Location-based social networks (LBSNs) are increasingly popular, allowing users to share their experiences at various Points of Interest (POIs). Recommending relevant POIs to users is crucial for enhancing their urban living experience, helping them discover new places and avoid wasted time searching or traveling. While graph deep learning methods have shown promise in this area by modeling user check-in behaviors as graphs, they struggle to capture long-range dependencies due to issues like over-smoothing in graph neural networks. Furthermore, the sheer scale of real-world LBSNs poses significant computational challenges for these models. This paper aims to overcome these limitations by proposing a more effective and scalable recommendation system.
Methods
The core of the proposed method is the spatiotemporal aware transformer (STAT). It builds upon the Transformer architecture, specifically its encoder, to learn POI representations.
- Geographically Aware Attention Mechanism: Traditional self-attention mechanisms in Transformers focus solely on semantic features. STAT enhances this by introducing a novel attention mechanism that explicitly incorporates geographical information. It calculates the distance between POI pairs and uses this as a bias to strengthen the attention matrix, reflecting Tobler's first law of geography ("Everything is related to everything else, but near things are more related than distant things"). This allows the model to better capture geographical influences on user preferences.
- Temporally Aware Sampling Strategy: A significant challenge with Transformer-based models is their quadratic computational complexity with respect to the number of nodes (POIs). To address this for large-scale LBSNs, STAT employs a temporally aware sampling strategy. This strategy samples relevant POIs based on check-in timestamps, treating these sampled POIs as a "sentence" for the attention mechanism. This reduces the computational cost from O(m²) to O(s²) (where 's' is the sample size) and enables mini-batch training, making the model scalable. It also helps filter out irrelevant POIs, improving effectiveness.
- User Preference Estimation: The paper proposes a time-aware method for estimating user preferences. Instead of simply summing the representations of visited POIs, it uses a sigmoid function to weigh recent check-ins more heavily, acknowledging that user preferences evolve over time.
Datasets
The authors evaluated their proposed STAT model on two widely used real-world datasets:
- Foursquare: Collected from the Foursquare social network, containing user check-in records from 2010-2014 in New York and Tokyo. It includes user, POI, timestamp, and POI category information, as well as user friendships.
- Gowalla: Collected from the Gowalla mobile social platform, containing check-in records from February 2009 to October 2010. Each record includes user, POI, and timestamp information.
For both datasets, users and POIs with fewer than 20 check-in records were removed. The data was partitioned chronologically, with the first 80% used for training and the remaining 20% for testing.
Key Insights and Ideas
- Geography Matters (A Lot): The paper strongly emphasizes that geographical proximity is a fundamental driver of user behavior in LBSNs. Their geographically aware attention mechanism effectively leverages this, outperforming methods that don't explicitly model spatial relationships.
- Time is Dynamic: User preferences are not static. The proposed time-aware user preference estimation method, which gives more weight to recent check-ins, significantly improves recommendation accuracy, highlighting the importance of temporal dynamics.
- Transformers Can Scale: The traditional computational burden of Transformers was a major hurdle for large-scale LBSNs. The temporally aware sampling strategy is a clever solution that makes Transformer architectures practical for this domain.
- Beyond Simple Graph Structures: The paper demonstrates that while graph-based methods are useful, they can be limited by fixed graph structures and issues like over-smoothing. Transformer's self-attention, when augmented, can capture more nuanced, long-range dependencies.
Key Results
STAT consistently outperformed all baseline methods across both the Foursquare and Gowalla datasets on precision, recall, and NDCG metrics. For instance, on the Foursquare dataset, STAT achieved a recall of 7.3% for k=5, compared to 6.8% for the next best baseline (GPR). The paper also shows that STAT's performance improves with a sampling size of s=40, indicating an optimal balance between information capture and noise reduction. The efficiency study showed that STAT has good scalability, with manageable running times and memory costs for large-scale datasets.
Limitations and Future Work
The authors acknowledge that STAT currently only considers temporal and geographical factors. They suggest that future work could involve developing a framework to jointly model other important factors, such as social information among users and POI category information, to further enhance user preference learning.
Summary updated