Traditional 3D generative models have historically been "object-centric." They excel at synthesizing isolated chairs, cars, or human faces against simplified backgrounds, as seen in early work like Visual Object Networks by Jun-Yan Zhu and Jiajun Wu. However, the real world is not a vacuum; it is a complex composition of multiple interacting entities, varied lighting, and expansive backgrounds.
The research direction of Compositional and Scalable 3D Scene Synthesis aims to move beyond individual assets toward the generation of entire, interactive environments—ranging from residential rooms to entire city blocks.
Decomposition as a Generative Prior
The core intuition behind scene-scale synthesis is that a complex environment should be treated as a collection of independent parts rather than a single, monolithic volume. This is known as compositional modeling.
In GIRAFFE, Michael Niemeyer and Andreas Geiger introduced the idea of representing scenes as compositional 3D feature fields. Instead of rendering a single image, the model generates individual feature fields for each object and the background, then composes them in 3D space. This allows for "interactive" generation: a user can move, rotate, or clone a specific car in a scene without affecting the rest of the environment.
Similarly, BlockGAN by Thu Nguyen-Phuoc and colleagues uses a 3D inductive bias to learn object-aware representations from unlabelled 2D images. By disentangling the identity and pose of "blocks" (objects), the model ensures that lighting and shadows remain consistent even as the scene layout changes. DisCoScene further refined this by using 3D bounding boxes as a "layout prior," enabling high-fidelity synthesis of multi-object scenes where entities are spatially disentangled.
Scaling Through Hierarchical Representations
As scenes grow from single rooms to urban environments, the computational cost of standard 3D representations (like dense voxels or high-resolution NeRFs) becomes prohibitive. Researchers have turned to hierarchical structures to manage this complexity.
LT3SD, developed by Quan Meng and Angela Dai, uses a "latent tree" representation. By factorizing a scene into a coarse-to-fine hierarchy, the model can generate virtually infinite 3D worlds patch-by-patch. This bypasses the "bounded volume" problem of earlier models like GRAF, which were often restricted to centered objects.
For even larger scales, such as autonomous driving environments, LSD-3D by Andrea Ramazzina utilizes geometry grounding to distill 2D diffusion knowledge into 3D Gaussian Splatting. This allows for the creation of massive, physically grounded driving scenes that support real-time rendering.
Lifting 2D Priors to 3D Environments
A major breakthrough in this field is the realization that we don't necessarily need massive 3D datasets to generate 3D scenes. Instead, we can "lift" the rich, semantic knowledge of 2D diffusion models into 3D space.
- Iterative Inpainting: Text2Room generates room-scale meshes by iteratively "lifting" 2D images into 3D using depth estimation and text-conditioned inpainting. As the virtual camera moves, the model fills in the "holes," creating a watertight, textured environment.
- Gaussian Distillation: Models like GaussianDreamer and SegmentDreamer use 3D Gaussian Splatting as a bridge. They take the structural consistency of 3D priors and refine them with the high-frequency detail of 2D diffusion models. Jiemin Fang and others demonstrated that this approach can generate complex assets in minutes, though scaling these to multi-object environments remains a primary challenge due to the "Janus problem" (multi-faced artifacts) and spatial alignment issues.
Beyond Visuals: Physics and Functionality
The latest frontier in scene synthesis is ensuring that generated environments are not just "pictures" but functional spaces. PhyScene by Baoxiong Jia and Siyuan Huang introduces physics-based guidance—such as collision avoidance and agent reachability—into the diffusion process. This ensures that a generated kitchen, for example, is actually navigable for a robot, bridging the gap between generative art and Embodied AI simulation.
By unifying perception, generation, and editing—as attempted in Uni3D-LLM—the field is moving toward a future where natural language can be used to architect entire, physically consistent 3D worlds.