Abstract
Accurate semantic annotation of Points of Interest (POIs) is fundamental to urban planning and location-based services. While existing methods often depend on sparse user-generated content (UGC), we propose M3PA, a multimodal framework designed to classify POIs using only their names, locations, and surrounding street view images. M3PA integrates a novel geographic language model, GeoBERT, with a distance-aware visual attention mechanism to capture both local visual contexts and spatial co-occurrence patterns. Our evaluation on large-scale datasets from Beijing and Jinan shows that M3PA consistently outperforms state-of-the-art baselines, providing a robust solution for annotating POIs in data-scarce environments.
Introduction
Point of Interest (POI) categorization is a cornerstone of modern digital maps. Traditional annotation techniques rely heavily on check-in histories and user reviews. However, these "User-Generated Content" (UGC) sources are frequently missing for newly established businesses or locations in less-populated areas. This "cold-start" problem necessitates models that can infer semantics from intrinsic and environmental data: the POI's name, its geographic coordinates, and the visual characteristics of its neighborhood.
Methodology: The M3PA Framework
M3PA operates through a two-step process: representation learning and evaluation. The framework processes urban data through two specialized pipelines—visual and textual—before fusing them for final classification.
Visual Representation Learning
The visual pipeline extracts features from street view images within a 100-meter radius of the target POI. Using a ResNet-based encoder fine-tuned to predict the category distribution of nearby POIs, the model learns to associate specific architectural styles or storefront "vibes" with urban semantics. A Geo Attention mechanism is applied to these features, weighting them based on their physical distance to the target POI to prioritize immediate surroundings.
Textual Representation with GeoBERT
To capture the semantic relationships between neighboring places, we developed GeoBERT. Unlike standard language models, GeoBERT processes "pseudo-sentences" composed of the target POI name and its nearest neighbors.
Crucially, GeoBERT incorporates a multidimensional continuous geographical encoding. This allows the Transformer architecture to account for spatial proximity, enabling the model to learn co-occurrence patterns (e.g., the likelihood of a "Cinema" being located near a "Dessert House").
Multimodal Fusion
The resulting visual and textual vectors are merged via an attention-based fusion layer. This step allows the model to dynamically weigh the importance of visual context versus textual names, which is particularly useful when names are ambiguous (e.g., "Sunshine Center").
Experimental Results
The model was validated using two extensive datasets from Amap and Baidu Street View:
- Haidian District (Beijing): 89,055 POIs across 248 categories, paired with 185,388 street view images.
- Lixia District (Jinan): 44,514 POIs across 140 categories, paired with 91,672 street view images.
M3PA achieved an accuracy of 69.43% in Haidian and 68.51% in Lixia, outperforming baselines such as BERT and ResNet. Ablation studies confirmed that both the visual context and the geographic embeddings in GeoBERT are critical for performance, with the multimodal approach showing significant gains in Macro-F1 scores for rare categories.
Conclusion
M3PA demonstrates that combining street-level imagery with spatial-textual modeling can effectively bridge the gap left by missing UGC. By treating urban space as a multimodal "language," the framework provides a scalable and accurate method for POI semantic annotation in diverse urban environments.
Summary updated