UrbanMLLM: Joint Learning of Cross-view Imagery for Urban Understanding
Lists
Only you see this
Abstract
Multimodal large language models (MLLMs) have exhibited remarkable capabilities for performing complex vision-language tasks in various domains. Currently, MLLMs based on urban imagery in urban studies are only developed focusing on remote sensing imagery. However, except for the macroscopic information from remote sensing imagery, effective urban understanding also requires detailed appearance information of urban zones from street-view imagery, which is largely overlooked by existing MLLMs. The primary challenges of developing such a versatile urban MLLM are twofold. Firstly, it needs a large-scale corpus with well-organized, cross-view urban imagery paired with corresponding text for cross-modal training. Secondly, traditional MLLMs typically learn image-text pairs independently, hard to support joint modeling of cross-view urban imagery. To address these challenges, in this work, we propose UrbanMLLM, a novel MLLM that jointly learns from remote sensing and street-view imagery to harness their complementary information. We first collect a large-scale dataset containing satellite-view and street-view imagery along with their geotags and annotated texts. Technically, we propose a brand MLLM architecture with a cross-view perceiver to explicitly connect visual information of cross-view urban imagery. We also introduce a novel pre-training paradigm based on structural interleaved urban image-text documents integrating satellite-view, street-view imagery and related textual descriptions. This approach encourages the model to implicitly learn the relationships between different types of urban imagery, enhancing the understanding in each domain. We evaluate our model on a comprehensive benchmark comprising 13 diverse urban understanding tasks across satellite-view, street-view, and cross-view domains. These tasks include scene classification, object reasoning, spatial relationship reasoning, geo-localization, landmark reasoning, and indicator prediction, providing a robust assessment of the model's capabilities. Extensive experiments demonstrate that UrbanMLLM achieves an average of 27.3\% and 25.5\% performance improvement compared with the best open-sourced and closed-sourced MLLMs, respectively. Moreover, we thoroughly study the impact of different pre-training data choices and model scales on performance, offering practical insights for effective MLLM design. The proposed UrbanMLLM offers a scalable and versatile solution for understanding urban environments.
BibTeX
@inproceedings{zhang2026urbanmllm,
author = {Xin Zhang and Tianjian Ouyang and Yu Shang and Qingmin Liao and Yong Li},
title = {{UrbanMLLM: Joint Learning of Cross-view Imagery for Urban Understanding}},
booktitle = {International Conference on Machine Learning},
year = {2026},
url = {https://openreview.net/forum?id=Av10FtirO4},
}Loading related content...