MuseCL: Predicting Urban Socioeconomic Indicators via Multi-Semantic Contrastive Learning

Xixian Yong, Xiao Zhou

Abstract

Predicting socioeconomic indicators within urban regions is crucial for fostering inclusivity, resilience, and sustainability in cities and human settlements. While pioneering studies have attempted to leverage multi-modal data for socioeconomic prediction, jointly exploring their underlying semantics remains a significant challenge. To address the gap, this paper introduces a Multi-Semantic Contrastive Learning (MuseCL) framework for fine-grained urban region profiling and socioeconomic prediction. Within this framework, we initiate the process by constructing contrastive sample pairs for street view and remote sensing images, capitalizing on the similarities in human mobility and Point of Interest (POI) distribution to derive semantic features from the visual modality. Additionally, we extract semantic insights from POI texts embedded within these regions, employing a pre-trained text encoder. To merge the acquired visual and textual features, we devise an innovative cross-modality-based attentional fusion module, which leverages a contrastive mechanism for integration. Experimental results across multiple cities and indicators consistently highlight the superiority of MuseCL, demonstrating an average improvement of 10% in $R^2$ compared to various competitive baseline models.

Introduction

Accurate, fine-grained data on socioeconomic indicators—such as population density, economic activity, and crime rates—is the cornerstone of effective urban planning and the pursuit of global Sustainable Development Goals. Traditionally, these metrics are gathered through community surveys and censuses, which are notoriously labor-intensive, expensive, and often outdated by the time they are published.

In recent years, researchers have turned to "big data" sources, including satellite imagery, Street View photos, and Point of Interest (POI) data, to map urban dynamics in real-time. However, a significant hurdle remains: multi-modal integration. Most existing models rely on Tobler’s First Law of Geography, which assumes that locations in close physical proximity are inherently similar. In modern urban environments, this assumption often fails; a financial district and a high-end residential area may be kilometers apart yet share more functional and socioeconomic similarities than a business district does with an adjacent industrial park. The MuseCL framework addresses this by moving beyond geographic proximity to explore the underlying functional semantics of urban regions.

The MuseCL Framework

The Multi-Semantic Contrastive Learning (MuseCL) framework optimizes urban representation through a three-tiered approach to data fusion and feature extraction.

1. Visual Semantic Extraction via Contrastive Learning

Rather than pairing images based on distance, MuseCL constructs contrastive sample pairs based on functional similarity. This allows the model to learn embeddings that reflect how a space is actually used:

  • Street View Imagery: These images are paired based on Population Mobility patterns (the flow of people in and out of a region), capturing the social and human activity levels of a neighborhood.
  • Remote Sensing (Satellite) Imagery: These are paired based on POI Distribution (the density and types of businesses), capturing the physical and land-use characteristics of the area.

A Triplet Loss function is employed to ensure that regions with similar functional profiles are pulled closer together in the embedding space, while functionally dissimilar regions are pushed apart.

2. Textual Semantic Incorporation

Visual data alone cannot always distinguish the "vibe" or specific economic status of a region. MuseCL incorporates textual data from POIs—including categories, ratings, and reviews—using a pre-trained text encoder. This allows the model to differentiate between superficially similar buildings, such as a cluster of luxury boutiques versus a row of wholesale warehouses, providing a layer of functional context that imagery might miss.

3. Attentional Fusion and Alignment

To create a unified "urban fingerprint," the framework utilizes a cross-modality attentional fusion module. Instead of simple concatenation, this module learns the relative importance of each modality (Street View, Satellite, or Text) for a given region. These features are then aligned using an InfoNCE loss mechanism, ensuring that the visual and textual representations of the same region are consistent and complementary.

Experimental Evaluation

The researchers evaluated MuseCL across three major global metropolises: Beijing (BJ), Shanghai (SH), and New York City (NY). The model was tested against several socioeconomic indicators, including:

  • Population Density (WorldPop)
  • Housing Density and Housing Prices
  • Mobility Counts
  • Crime Rates (specifically in NYC)

Key Results

  • Performance Gains: MuseCL consistently outperformed state-of-the-art baselines like Urban2vec and Tile2vec. In Beijing, the model achieved an average $R^2$ improvement of 10%, with specific indicators seeing boosts of up to 20%.
  • Robustness: The model proved adaptable to different urban morphologies, performing well in both the hexagonal grid structures of Beijing and the dense, block-based layout of New York City.
  • Ablation Studies: Experiments confirmed that removing any single modality (POI, Street View, or Satellite) resulted in a performance drop, validating the necessity of the multi-semantic approach.

Discussion and Insights

The success of MuseCL highlights that function over proximity is a superior paradigm for urban modeling. By using human mobility and POI types as "anchors," the model can recognize a "financial district signature" or a "residential profile" regardless of where those areas are located geographically.

Furthermore, the researchers used Principal Component Analysis (PCA) to visualize the learned embeddings, revealing clear clusters that represent different stages of urban development: from underdeveloped outskirts to emerging suburban villages and highly urbanized commercial hubs. This visualization provides a powerful tool for policymakers to track a city's "growth direction."

Conclusion and Future Work

MuseCL represents a significant step forward in fine-grained urban profiling. By successfully fusing visual and textual semantics through contrastive learning, it provides a more accurate and scalable alternative to traditional survey methods.

Future research will focus on the temporal dimension. While the current model provides a high-fidelity snapshot of a city, cities are dynamic organisms. Incorporating time-series data to track how socioeconomic indicators evolve over months or years will be essential for long-term urban resilience and sustainability planning. Additionally, addressing data scarcity in developing regions remains a priority to ensure these tools can be applied globally.

Summary updated

Abstract

Predicting socioeconomic indicators within urban regions is crucial for fostering inclusivity, resilience, and sustainability in cities and human settlements. While pioneering studies have attempted to leverage multi-modal data for socioeconomic prediction, jointly exploring their underlying semantics remains a significant challenge. To address the gap, this paper introduces a Multi-Semantic Contrastive Learning (MuseCL) framework for fine-grained urban region profiling and socioeconomic prediction. Within this framework, we initiate the process by constructing contrastive sample pairs for street view and remote sensing images, capitalizing on the similarities in human mobility and Point of Interest (POI) distribution to derive semantic features from the visual modality. Additionally, we extract semantic insights from POI texts embedded within these regions, employing a pre-trained text encoder. To merge the acquired visual and textual features, we devise an innovative cross-modality-based attentional fusion module, which leverages a contrastive mechanism for integration. Experimental results across multiple cities and indicators consistently highlight the superiority of MuseCL, demonstrating an average improvement of 10% in $R^2$ compared to various competitive baseline models. The code of this work is publicly available at https://github.com/XixianYong/MuseCL.
BibTeX
@inproceedings{yong2024musecl,
  author = {Xixian Yong and Xiao Zhou},
  title = {{MuseCL: Predicting Urban Socioeconomic Indicators via Multi-Semantic Contrastive Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=PVhOYg4hEY},
}
Figure