MuseCL: Predicting Urban Socioeconomic Indicators via Multi-Semantic Contrastive Learning

Xixian Yong, Xiao Zhou

Abstract

Predicting socioeconomic indicators within urban regions is crucial for fostering inclusivity, resilience, and sustainability in cities and human settlements. While pioneering studies have attempted to leverage multi-modal data for socioeconomic prediction, jointly exploring their underlying semantics remains a significant challenge. To address the gap, this paper introduces a Multi-Semantic Contrastive Learning (MuseCL) framework for fine-grained urban region profiling and socioeconomic prediction. Within this framework, we initiate the process by constructing contrastive sample pairs for street view and remote sensing images, capitalizing on the similarities in human mobility and Point of Interest (POI) distribution to derive semantic features from the visual modality. Additionally, we extract semantic insights from POI texts embedded within these regions, employing a pre-trained text encoder. To merge the acquired visual and textual features, we devise an innovative cross-modality-based attentional fusion module, which leverages a contrastive mechanism for integration. Experimental results across multiple cities and indicators consistently highlight the superiority of MuseCL, demonstrating an average improvement of 10% in R2 compared to various competitive baseline models.

Introduction

Urban planners and policymakers require accurate, fine-grained data on socioeconomic indicators—such as population density, housing costs, and business activity—to build sustainable cities. Traditionally, these metrics are derived from census surveys, which are often expensive, infrequent, and slow to produce.

While recent advancements in AI have enabled the analysis of cities through satellite and street-view imagery, two major hurdles persist. First, most models are constrained by "Tobler’s Law," assuming that geographically proximal regions are inherently similar. However, in modern urban environments, a commercial hub in one district may be more semantically similar to a hub miles away than to an adjacent residential park. Second, effectively fusing diverse data modalities—visual imagery, textual POI descriptions, and behavioral mobility patterns—into a cohesive neighborhood profile remains a complex task.

Illustration of urban mobility and POI distribution as semantic bridges between geographically distant regions.

Methodology: The MuseCL Framework

The Multi-Semantic Contrastive Learning (MuseCL) framework addresses these challenges by creating mathematical "embeddings" for urban regions through a multi-step contrastive approach.

Semantic Contrastive Learning

Rather than relying on geographic proximity, MuseCL constructs contrastive pairs based on functional and behavioral similarities:

  • Mobility Flow: Street-view images are paired if they exhibit similar patterns of human inflow and outflow.
  • POI Distribution: Remote sensing (satellite) images are paired if they share similar distributions of business types and services (e.g., industrial vs. commercial clusters).

Multi-Modal Integration

The framework employs specialized encoders for different data types, as shown in the architecture below:

  • Visual Encoders: Separate encoders for street view and remote sensing imagery, updated via triplet loss based on mobility and POI types.
  • POI Encoder: Extracts semantic insights from POI names and categories to capture the functional "vibe" of a neighborhood.
  • Attentional Fusion Module: A cross-modality attention mechanism merges these features. It utilizes an InfoNCE loss function to align visual and textual representations, ensuring the final profile captures both the appearance and the actual function of a region.
The MuseCL framework architecture, integrating street view, remote sensing, and POI data through contrastive learning and attentional fusion.

Experimental Evaluation

The study evaluated MuseCL across three global metropolises: Beijing (BJ), Shanghai (SH), and New York (NY). Data sources included imagery from Baidu and Google Maps, contextual data from OpenStreetMap, and ground truth indicators like WorldPop for population and Lianjia for housing prices.

Key Results

  • Superior Accuracy: MuseCL consistently outperformed baseline models like ResNet-18 and Urban2vec. In Beijing, the model achieved $R^2$ improvements ranging from 4.81% to 20.70% across various indicators.
  • Precision in POI Prediction: The model demonstrated exceptional performance in predicting the "Number of POIs" in Beijing, reaching an $R^2$ of 0.9471.
  • Cross-City Robustness: MuseCL maintained high performance across different urban morphologies, from the sprawling structure of Beijing to the compact grid of New York.
  • Latent Space Clustering: Visualization of the model's results showed that it naturally clusters regions into logical categories—such as "commercial hubs" or "suburban residential"—without explicit supervision.

Conclusion and Future Directions

MuseCL demonstrates the power of using human mobility and functional similarity as semantic bridges for urban profiling. By moving beyond simple geographic proximity, the framework provides a more nuanced understanding of city dynamics.

Future work will focus on incorporating time-series data to capture the dynamic evolution of cities. Additionally, addressing data dependency in regions where high-quality street-view or mobility data is sparse remains a priority for global applicability. Finally, exploring the impact of region size (e.g., 500m vs. 1km grids) will help refine the granularity of socioeconomic predictions.

The code for this work is available at https://github.com/XixianYong/MuseCL.

Summary updated

Abstract

Predicting socioeconomic indicators within urban regions is crucial for fostering inclusivity, resilience, and sustainability in cities and human settlements. While pioneering studies have attempted to leverage multi-modal data for socioeconomic prediction, jointly exploring their underlying semantics remains a significant challenge. To address the gap, this paper introduces a Multi-Semantic Contrastive Learning (MuseCL) framework for fine-grained urban region profiling and socioeconomic prediction. Within this framework, we initiate the process by constructing contrastive sample pairs for street view and remote sensing images, capitalizing on the similarities in human mobility and Point of Interest (POI) distribution to derive semantic features from the visual modality. Additionally, we extract semantic insights from POI texts embedded within these regions, employing a pre-trained text encoder. To merge the acquired visual and textual features, we devise an innovative cross-modality-based attentional fusion module, which leverages a contrastive mechanism for integration. Experimental results across multiple cities and indicators consistently highlight the superiority of MuseCL, demonstrating an average improvement of 10% in R2 compared to various competitive baseline models. The code of this work is publicly available at https://github.com/XixianYong/MuseCL.
BibTeX
@inproceedings{yong2024musecl,
  author = {Xixian Yong and Xiao Zhou},
  title = {{MuseCL: Predicting Urban Socioeconomic Indicators via Multi-Semantic Contrastive Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=PVhOYg4hEY},
}
Figure