TL;DR
This paper introduces Tool4POI, a novel framework that augments Large Language Models (LLMs) with external tools for next Point-of-Interest (POI) recommendation. By enabling LLMs to retrieve and reason over external information, Tool4POI overcomes limitations of existing methods, particularly in recommending "out-of-history" locations and scaling to large POI spaces, achieving significant performance improvements.
What Problem Does This Paper Address?
Recommending the next place a user might want to visit (Next POI Recommendation) is crucial for location-based services. While LLMs have shown promise due to their reasoning capabilities, current LLM-based approaches struggle with two main issues:
- Out-of-History (OOH) Scenarios: LLMs heavily rely on the user's past check-in history. If the target POI isn't in this history, LLMs perform poorly. This is a significant problem, as OOH recommendations are common (over 30% in real-world datasets) and essential for discovering new places and adding diversity to recommendations.
- Scalability to Large POI Spaces: LLMs have limited context windows, making it impractical to feed them information about every possible POI in a city (which can number in the hundreds of thousands). This restricts their ability to perform "open-set" recommendations, where the model can suggest any POI, not just those it has seen before.
Methods
Tool4POI tackles these challenges with a three-module framework:
- Preference Extraction Module: This module uses an LLM to analyze a user's long-term historical check-in data. It extracts preferences across dimensions like region, category, and time. To better capture spatial relationships, it converts geographical coordinates into Google Maps' Plus Codes, which group nearby locations.
- Tool-Augmented Candidate Retrieval Module: This is the core innovation. Instead of relying solely on the LLM's internal knowledge, this module allows the LLM to interact with a set of external "tools." These tools can query POI information, filter POIs by category or region, find potentially relevant POIs based on collaborative filtering, and sort POIs by distance. This multi-turn interaction allows the LLM to dynamically search and refine a candidate set of POIs, effectively overcoming the context window limitation and enabling exploration of POIs beyond the user's history.
- Reranking Module: After retrieving a candidate set, this module uses the LLM to reorder the candidates based on the user's recent check-in behavior. This helps capture short-term interests and current intent, ensuring the final recommendations are relevant to the user's immediate context.
A key aspect of Tool4POI is its "plug-and-play" nature; it requires no task-specific fine-tuning and can work with off-the-shelf LLMs.
Datasets
The authors evaluated Tool4POI on three widely used real-world datasets:
- Foursquare-NYC: Data from New York City.
- TKY: Data from Tokyo.
- Gowalla-CA: Data from California.
These datasets vary in size and sparsity, with Gowalla-CA being particularly sparse, making it a challenging testbed for OOH recommendation scenarios.
Key Insights and Ideas
- LLMs as "Tool Users": The paper demonstrates that LLMs can be effectively augmented with external tools to perform complex reasoning and information retrieval tasks, moving beyond their inherent limitations.
- Bridging the OOH Gap: The tool-augmented retrieval is crucial for recommending POIs that a user has never visited before, a significant limitation of standard LLM approaches.
- Scalability through Iteration: The multi-turn retrieval process allows LLMs to efficiently explore vast POI spaces without needing to encode all information in the prompt, making it scalable to real-world applications.
- Complementary Strengths: The preference extraction, retrieval, and reranking modules work synergistically. Long-term preferences guide retrieval, while recent behavior refines the final ranking.
Key Results
- Tool4POI substantially outperforms state-of-the-art baselines across all datasets and metrics.
- On challenging OOH scenarios, where existing methods often achieve zero accuracy, Tool4POI achieves remarkable performance, reaching up to 40% accuracy on Acc@10.
- Overall, Tool4POI delivers average improvements of 20% on Acc@5 and 30% on Acc@10 compared to baselines.
- Even with a smaller LLM backbone (3B parameters), Tool4POI outperforms larger, fine-tuned LLM methods on top-k metrics, highlighting the efficiency of its tool-augmented approach.
Limitations and Future Work
- LLM Backbone Dependency: While Tool4POI is efficient, its performance is still influenced by the underlying LLM's capabilities. Larger LLMs generally lead to better performance, particularly in tool invocation and reranking.
- Tool Design: The effectiveness of the framework relies on the design and quality of the external tools. Future work could explore more sophisticated tool sets or dynamic tool generation.
- Interpretability: While the tool-use provides some interpretability, further work could enhance the transparency of the LLM's decision-making process.
- Real-time Performance: The paper focuses on offline evaluation; future work could investigate optimizing Tool4POI for real-time recommendation scenarios.
Summary updated