Abstract
Semantic annotation for points of interest (POIs) is the process of annotating a POI with a category label, which facilitates many services related to POIs, such as POI search and recommendation. Most existing solutions extract features related to POIs from abundant user-generated content (UGC) data (e.g., check-ins and user comments). However, such data are often difficult to obtain, especially for newly created POIs. In this paper, we aim to explore semantic annotation for POIs with limited information such as POI (place) names and geographic locations. Additionally, we have found that street view images provide extensive visual clues about POI attributes and could be an essential supplement to limited information. To this end, we propose a novel multimodal model for POI semantic annotation, namely M3PA, which achieves enhanced semantic annotation through fusing a POI’s textual and visual representations. Extensive experiments with POI data from Amap validate the effectiveness of M3PA for POI semantic annotation, compared with several competitive baselines.
Introduction
Semantic annotation—the process of assigning a category (like "Restaurant" or "Park") to a location—is the backbone of modern map services and urban planning. Traditionally, models rely on "digital breadcrumbs" left by users: check-in logs, ratings, and detailed descriptions. However, this creates a significant "cold start" problem. For a newly opened shop or a remote landmark, these data sources are non-existent. While we might have the POI's name and coordinates, that is often insufficient for a machine to distinguish between ambiguous names (e.g., "Healthy Fresh" could be a salad bar or a gym). This research argues that to understand a place, we must mimic human perception: observing the physical environment via street view images and considering the neighborhood context of surrounding businesses.
Methodology: The M3PA Model
The Multimodal Model for POI Semantic Annotation (M3PA) consists of two primary branches designed to capture the visual and textual essence of a location.
Visual Representation
The model utilizes a pre-trained image encoder (based on ResNet) to extract features from street view images within a 100-meter buffer of the POI. A core innovation is the Geographic Attention mechanism. Recognizing that images closer to the POI are typically more representative than distant ones, the model assigns weights to visual features based on their geodesic distance from the target.
Textual Representation (GeoBERT)
To handle text, the authors developed GeoBERT, a modified BERT architecture. Instead of standard sentences, it processes "pseudo-sentences" consisting of the target POI name followed by its neighbors' names. Crucially, Geographical Embeddings map 2D coordinates into high-dimensional vectors, allowing the transformer layers to learn spatial relationships like proximity and orientation directly.
Finally, these visual and textual features are fused using an attention-based mechanism to predict the most likely category label.
Key Insights: Visual and Spatial Correlation
A fundamental premise of this work is that a POI's identity is intrinsically linked to its appearance and its neighbors. The authors demonstrated a strong Visual Correlation: POIs with similar street view characteristics are statistically more likely to belong to the same category. This validates the use of computer vision as a proxy for urban functional analysis.
Furthermore, the "Neighborhood Effect" suggests that spatial co-occurrence patterns (e.g., a pharmacy near a hospital) are vital. By training GeoBERT on sequences of nearby names, the model captures these localized urban semantics.
Datasets and Results
The study utilized two primary datasets from AMap (a major Chinese mapping service):
- Haidian District (Beijing): 89,055 POIs across 248 categories.
- Lixia District (Jinan): 44,514 POIs across 140 categories.
Over 277,000 street view images from Baidu Maps were gathered to provide the visual component. M3PA achieved state-of-the-art results across both datasets:
- Accuracy: In the Haidian dataset, M3PA reached 69.43% accuracy, a 2.03% improvement over the best baseline. In Lixia, it reached 68.51%, a 4.36% improvement.
- Macro-F1 & MRR: The model showed even stronger gains in Macro-F1 (up to 5.49%), indicating superior performance in identifying rare or niche POI categories.
- Multimodal Synergy: Ablation studies confirmed that while textual names are strong predictors, street view images provide a significant boost, acting as a "tie-breaker" when names are ambiguous.
Limitations and Future Work
While powerful, the model requires street view coverage within 100 meters, which may be missing in rural or rapidly developing areas. Additionally, static images do not capture temporal dynamics (e.g., a building that changes function at night). Future research may explore incorporating urban mobility data or satellite imagery to further refine these urban semantic models.
Summary updated