TL;DR
CityDreamer is a novel compositional generative model designed to build realistic, unbounded 3D cities by separating the generation of distinct building instances from background elements like roads and parks. By tailoring specific neural fields to the unique texture patterns of buildings (periodic) versus backgrounds (irregular), it achieves state-of-the-art 3D consistency and realism while enabling localized editing.
What Problem Does This Paper Address?
3D city generation is notoriously difficult because humans are highly attuned to structural distortions in urban environments—a warped skyscraper looks far worse than a warped tree. Furthermore, unlike natural scenes where objects of the same class (e.g., trees or grass) have relatively uniform textures, buildings exhibit a massive variety of architectural styles, heights, and textures despite sharing the same semantic label.
Existing 3D generation methods often treat cities as monolithic scenes or apply a single neural field to the entire environment. This leads to severe structural distortions, blurry building façades, and a lack of visual diversity. CityDreamer aims to generate infinitely expandable ("unbounded") 3D cities that maintain sharp, realistic geometries and diverse architectural styles.
Methods
The authors propose a compositional pipeline that explicitly decouples the city into background "stuff" and building "instances." The generation follows a four-step process:
- Unbounded Layout Generator: Uses a VQVAE and an autoregressive transformer (MaskGIT) to generate and endlessly extrapolate 2D semantic maps and height fields, which are then extruded into a 3D layout volume.
- City Background Generator: Handles background elements like roads, water, and greenery. It parameterizes the scene using a global encoder and a generative hash grid, which excels at capturing irregular, natural textures.
- Building Instance Generator: Identifies individual buildings via connected components. Crucially, instead of a hash grid, it uses a local encoder paired with periodic positional encoding (SinCos). This specific design choice leverages the inherent periodicity of building façades (windows, floors) to prevent structural warping.
- Compositor: Blends the separately rendered background and building instances using generated binary masks into a final cohesive image.
Datasets
To train and evaluate CityDreamer, the authors introduced the CityGen Datasets suite:
- OSM Dataset: Sourced from OpenStreetMap, containing rasterized semantic maps and height fields across 80 cities worldwide, covering over 6,000 km².
- GoogleEarth Dataset: Created using Google Earth Studio, featuring 24,000 real-world city images across 400 orbit trajectories in New York City, complete with automatically generated semantic and building instance segmentation masks.
Key Insights and Ideas
- Texture-Specific Parameterization: The core "aha" moment is that different elements of a city require different mathematical representations. Background "stuff" has stochastic, irregular textures best served by hash grids, while building façades have highly regular, repeating patterns best captured by periodic (SinCos) positional encodings.
- Instance-Level Disentanglement: Treating buildings as distinct, individual instances rather than a single semantic class prevents the model from averaging out architectural details, dramatically reducing structural distortions.
Key Results
- Quantitative Superiority: CityDreamer significantly outperforms state-of-the-art baselines like SGAM, PersistentNature, and SceneDreamer across multiple metrics on the GoogleEarth dataset. It achieved a Frechet Inception Distance (FID) of 97.38 (compared to SceneDreamer's 213.56) and a drastically lower Camera Error (CE) of 0.060, proving its superior multi-view consistency.
- Qualitative Excellence: As shown in the qualitative comparisons, CityDreamer produces sharp, structurally sound buildings and diverse city layouts that maintain strict multi-view consistency, avoiding the severe distortions seen in other models.
- Downstream Applications: Because the architecture is compositional, it natively supports localized editing—such as adjusting a single building's height or swapping styles between two buildings without altering the rest of the city—as well as realistic relighting (via Lambertian shading and shadow mapping).
Limitations and Future Work
- No Concave Geometries: Because the city layout relies on extruding a 2D height field into voxels, it cannot represent complex concave structures like tunnels, bridges, or caves.
- Inference Efficiency: Generating building instances individually during the inference phase increases the overall computational overhead. The authors note that exploring ways to reduce this inference cost is a promising direction for future work.
Summary updated