CityDreamer: Compositional Generative Model of Unbounded 3D Cities

Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei Liu

TL;DR

CityDreamer is a novel compositional generative model designed to build realistic, unbounded 3D cities by separating the generation of distinct building instances from background elements like roads and parks. By tailoring specific neural fields to the unique texture patterns of buildings (periodic) versus backgrounds (irregular), it achieves state-of-the-art 3D consistency and realism while enabling localized editing.

What Problem Does This Paper Address?

3D city generation is notoriously difficult because humans are highly attuned to structural distortions in urban environments—a warped skyscraper looks far worse than a warped tree. Furthermore, unlike natural scenes where objects of the same class (e.g., trees or grass) have relatively uniform textures, buildings exhibit a massive variety of architectural styles, heights, and textures despite sharing the same semantic label.

Existing 3D generation methods often treat cities as monolithic scenes or apply a single neural field to the entire environment. This leads to severe structural distortions, blurry building façades, and a lack of visual diversity. CityDreamer aims to generate infinitely expandable ("unbounded") 3D cities that maintain sharp, realistic geometries and diverse architectural styles.

Methods

The authors propose a compositional pipeline that explicitly decouples the city into background "stuff" and building "instances." The generation follows a four-step process:

  1. Unbounded Layout Generator: Uses a VQVAE and an autoregressive transformer (MaskGIT) to generate and endlessly extrapolate 2D semantic maps and height fields, which are then extruded into a 3D layout volume.
  2. City Background Generator: Handles background elements like roads, water, and greenery. It parameterizes the scene using a global encoder and a generative hash grid, which excels at capturing irregular, natural textures.
  3. Building Instance Generator: Identifies individual buildings via connected components. Crucially, instead of a hash grid, it uses a local encoder paired with periodic positional encoding (SinCos). This specific design choice leverages the inherent periodicity of building façades (windows, floors) to prevent structural warping.
  4. Compositor: Blends the separately rendered background and building instances using generated binary masks into a final cohesive image.
Overview of the CityDreamer architecture showing the layout generator, background generator, building instance generator, and compositor.

Datasets

To train and evaluate CityDreamer, the authors introduced the CityGen Datasets suite:

  • OSM Dataset: Sourced from OpenStreetMap, containing rasterized semantic maps and height fields across 80 cities worldwide, covering over 6,000 km².
  • GoogleEarth Dataset: Created using Google Earth Studio, featuring 24,000 real-world city images across 400 orbit trajectories in New York City, complete with automatically generated semantic and building instance segmentation masks.

Key Insights and Ideas

  • Texture-Specific Parameterization: The core "aha" moment is that different elements of a city require different mathematical representations. Background "stuff" has stochastic, irregular textures best served by hash grids, while building façades have highly regular, repeating patterns best captured by periodic (SinCos) positional encodings.
  • Instance-Level Disentanglement: Treating buildings as distinct, individual instances rather than a single semantic class prevents the model from averaging out architectural details, dramatically reducing structural distortions.

Key Results

  • Quantitative Superiority: CityDreamer significantly outperforms state-of-the-art baselines like SGAM, PersistentNature, and SceneDreamer across multiple metrics on the GoogleEarth dataset. It achieved a Frechet Inception Distance (FID) of 97.38 (compared to SceneDreamer's 213.56) and a drastically lower Camera Error (CE) of 0.060, proving its superior multi-view consistency.
  • Qualitative Excellence: As shown in the qualitative comparisons, CityDreamer produces sharp, structurally sound buildings and diverse city layouts that maintain strict multi-view consistency, avoiding the severe distortions seen in other models.
Qualitative comparison of CityDreamer against baseline models, demonstrating sharper building details and better structural consistency.
  • Downstream Applications: Because the architecture is compositional, it natively supports localized editing—such as adjusting a single building's height or swapping styles between two buildings without altering the rest of the city—as well as realistic relighting (via Lambertian shading and shadow mapping).

Limitations and Future Work

  • No Concave Geometries: Because the city layout relies on extruding a 2D height field into voxels, it cannot represent complex concave structures like tunnels, bridges, or caves.
  • Inference Efficiency: Generating building instances individually during the inference phase increases the overall computational overhead. The authors note that exploring ways to reduce this inference cost is a promising direction for future work.

Summary updated

Abstract

3D city generation is a desirable yet challenging task, since humans are more sensitive to structural distortions in urban environments. Additionally, generating 3D cities is more complex than 3D natural scenes since buildings, as objects of the same class, exhibit a wider range of appearances compared to the relatively consistent appearance of objects like trees in natural scenes. To address these challenges, we propose \textbf{CityDreamer}, a compositional generative model designed specifically for unbounded 3D cities. Our key insight is that 3D city generation should be a composition of different types of neural fields: 1) various building instances, and 2) background stuff, such as roads and green lands. Specifically, we adopt the bird's eye view scene representation and employ a volumetric render for both instance-oriented and stuff-oriented neural fields. The generative hash grid and periodic positional embedding are tailored as scene parameterization to suit the distinct characteristics of building instances and background stuff. Furthermore, we contribute a suite of CityGen Datasets, including OSM and GoogleEarth, which comprises a vast amount of real-world city imagery to enhance the realism of the generated 3D cities both in their layouts and appearances. CityDreamer achieves state-of-the-art performance not only in generating realistic 3D cities but also in localized editing within the generated cities.
BibTeX
@inproceedings{xie2023citydreamer,
  author = {Haozhe Xie and Zhaoxi Chen and Fangzhou Hong and Ziwei Liu},
  title = {{CityDreamer: Compositional Generative Model of Unbounded 3D Cities}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure