UrbanWorld: An Urban World Model for 3D City Generation

Yu Shang, Yuming Lin, Yu Zheng, Fan Hangyu, Jingtao Ding, Jie Feng, Jiansheng Chen, Tian Li, Yong LiSubmission to ICLR 2025 · Rejected

Abstract

Cities, as the essential environment of human life, encompass diverse physical elements such as buildings, roads and vegetation, which continuously interact with dynamic entities like people and vehicles. Crafting realistic, interactive 3D urban environments is essential for nurturing AGI systems and constructing AI agents capable of perceiving, decision-making, and acting like humans in real-world environments. However, creating high-fidelity 3D urban environments usually entails extensive manual labor from designers, involving intricate detailing and representation of complex urban elements. Therefore, accomplishing this automatically remains a longstanding challenge. Toward this problem, we propose UrbanWorld, the first generative urban world model that can automatically create a customized, realistic and interactive 3D urban world with flexible control conditions. Specifically, we design a progressive diffusion-based rendering method to produce 3D urban assets with high-quality textures. Moreover, we propose a specialized urban multimodal large language model (Urban MLLM) trained on realistic street-view image-text corpus to supervise and guide the generation process. UrbanWorld incorporates four key stages in the generation pipeline: flexible 3D layout generation from OSM data or urban layout with semantic and height maps, urban scene design with Urban MLLM, controllable urban asset rendering via progressive 3D diffusion, and MLLM-assisted scene refinement. We conduct extensive quantitative analysis on five visual metrics, demonstrating that UrbanWorld achieves state-of-the-art generation realism. Next, we provide qualitative results about the controllable generation capabilities of UrbanWorld using both textual and image-based prompts. Lastly, we verify the interactive nature of these environments by showcasing the agent perception and navigation within the created environments. We contribute UrbanWorld as an open-source tool available at https://github.com/Urban-World/UrbanWorld.
BibTeX
@misc{shang2024urbanworld,
  author = {Yu Shang and Yuming Lin and Yu Zheng and Fan Hangyu and Jingtao Ding and Jie Feng and Jiansheng Chen and Tian Li and Yong Li},
  title = {{UrbanWorld: An Urban World Model for 3D City Generation}},
  year = {2024},
  eprint = {2407.11965},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.11965},
}
Figure