TL;DR
This paper introduces UrbanWorld, the first generative model capable of automatically creating realistic, interactive, and customizable 3D urban environments. By combining a specialized multimodal large language model (Urban MLLM) with diffusion-based rendering, UrbanWorld addresses the long-standing challenge of manual labor in 3D city generation, paving the way for more advanced AI agent development.
What Problem Does This Paper Address?
Creating realistic and interactive 3D urban environments is crucial for training AI agents that can operate in the real world, but it's an incredibly labor-intensive process. Traditional methods rely heavily on manual design, requiring intricate detailing of buildings, roads, and vegetation. While recent advancements in generative AI have shown promise, existing approaches often produce static scenes, lack fine-grained control, or fail to generate novel assets. This leaves a significant gap in the ability to create dynamic, customizable, and truly "world-like" environments for embodied AI research.
Methods
UrbanWorld employs a novel "map-design-render-refine" pipeline to tackle this challenge:
- Flexible 3D Urban Layout Generation: It takes various input formats, including OpenStreetMap (OSM) data and semantic/depth maps, to automatically generate untextured 2.5D to 3D urban layouts. This step ensures the foundational structure of the city is established.
- Urban MLLM-Empowered Scene Design: A key innovation is the Urban MLLM, a multimodal large language model fine-tuned on extensive urban street-view imagery and text. This model acts as a virtual designer, generating detailed textual descriptions for urban assets based on user prompts, ensuring visual coherence and adherence to instructions.
- Controllable Diffusion-based Urban Asset Texture Rendering: To achieve high-fidelity textures, the system uses a diffusion-based approach. It renders individual urban elements (buildings, roads, vegetation, water) using depth-aware diffusion models, controlled by the textual descriptions from the Urban MLLM or reference images. This is followed by an UV texture inpainting process, guided by UV position maps, to fill in any missing details and ensure completeness.
- MLLM-Assisted Scene Refinement: Finally, the Urban MLLM scrutinizes the generated environment for texture inconsistencies and provides suggestions for refinement. This iterative process, mimicking human design workflows, further enhances the visual quality and realism of the final urban scene.
Datasets
The paper mentions using a curated dataset of approximately 300K urban street-view images from Google Maps, augmented with GPT-4 generated textual descriptions, to fine-tune the Urban MLLM. For evaluation, they use a set of 1000 real street-view images randomly sampled from Google Maps for metrics like FID and KID.
Key Insights and Ideas
- The Power of Specialized LLMs: The Urban MLLM is a critical component, demonstrating that fine-tuning LLMs on domain-specific data (urban environments) can unlock sophisticated scene understanding and generation capabilities, mimicking human designers.
- Element-wise Rendering for Quality: Instead of rendering entire scenes at once, breaking down the process into element-wise rendering and then composing them significantly improves texture quality and reduces mismatches.
- Iterative Refinement is Key: The MLLM-assisted refinement loop is crucial for achieving state-of-the-art realism, highlighting that even advanced generative models benefit from a human-like iterative feedback mechanism.
- Bridging the Gap to Embodied AI: UrbanWorld's focus on creating interactive environments is a significant step towards enabling more realistic training and evaluation of embodied AI agents.
Key Results
UrbanWorld demonstrates state-of-the-art performance across several quantitative metrics:
- Realism: It achieves significant improvements in Frechét Inception Distance (FID) and Kernel Inception Distance (KID) compared to existing methods, indicating more realistic generated scenes. For example, UrbanWorld (image) shows a 5.4% and 26.7% improvement in FID and KID, respectively, over the most competitive baseline.
- Geometry Accuracy: The model achieves a 44.2% improvement in depth error (DE), showcasing its ability to preserve geometric consistency.
- Diversity: UrbanWorld exhibits a 10.4% improvement in homogeneity index (HI), meaning it can generate more diverse urban environments, moving beyond the limitations of training data styles.
- Controllability: Qualitative results show impressive control over style (e.g., neoclassical, Bauhaus) and functional scenes (residential, CBD, park) using both text and image prompts.
- Preference Score: UrbanWorld-generated scenes received higher preference scores from GPT-4, suggesting superior overall quality and realism.
Limitations and Future Work
The paper acknowledges that while UrbanWorld can create new assets, it relies on a diffusion-based renderer. Future work could explore more advanced asset generation techniques. The authors also contribute UrbanWorld as an open-source tool, suggesting its potential for broad research in embodied intelligence and AI agents, implying that further community contributions and extensions are anticipated. The paper also mentions that some baselines were not included due to their code not being open-source, which is a common limitation in comparative studies.
Summary updated