Abstract
The generation of 3D scenes from user-specified conditions offers a promising avenue for alleviating the production burden in 3D applications. Previous studies required significant effort to realize the desired scene, owing to limited control conditions. We propose a method for controlling and generating 3D scenes under multimodal conditions using partial images, layout information represented in the top view, and text prompts. Combining these conditions to generate a 3D scene involves significant difficulties: (1) the creation of large datasets, (2) reflection on the interaction of multimodal conditions, and (3) domain dependence of the layout conditions. We decompose the process of 3D scene generation into 2D image generation from the given conditions and 3D scene generation from 2D images. 2D image generation is achieved by fine-tuning a pretrained text-to-image model with a small artificial dataset of partial images and layouts, and 3D scene generation is achieved by layout-conditioned depth estimation and neural radiance fields (NeRF), thereby avoiding the creation of large datasets. The use of a common representation of spatial information using 360-degree images allows for the consideration of multimodal condition interactions and reduces the domain dependence of the layout control. The experimental results qualitatively and quantitatively demonstrated that the proposed method can generate 3D scenes in diverse domains, from indoor to outdoor, according to multimodal conditions.
1. Introduction
High-quality 3D scene generation remains a significant challenge in computer vision, primarily due to the trade-off between creative freedom and precise control. Existing methods often suffer from input limitations, typically relying on a single modality like text or a single image. While text provides stylistic guidance, it lacks spatial precision; conversely, layouts offer geometric control but lack visual richness. Furthermore, the scarcity of large-scale 3D datasets and the domain rigidity of current models—often restricted to specific indoor or outdoor environments—hinder broad applicability.
MaGRITTe (Manipulative and Generative 3D Realization from Image, Topview and Text) addresses these challenges by introducing a multimodal framework that accepts a partial image, a top-view layout, and a text prompt. By leveraging 360-degree equirectangular projection (ERP) as a bridge, the system harmonizes these disparate inputs into a unified spatial representation.
2. Methodology
The MaGRITTe pipeline decomposes the 3D generation task into two primary phases: 2D panorama generation and 3D scene lifting. This approach bypasses the need for massive 3D training datasets by utilizing pre-trained 2D diffusion models.
2.1 ERP Conversion and RGB Generation
All spatial inputs are first converted into Equirectangular Projection (ERP) format. This allows the model to process floor plans and perspective images within a single coordinate system. A pre-trained Stable Diffusion model, enhanced with ControlNet, is fine-tuned on small synthetic datasets. The model receives the ERP-formatted partial image, a coarse depth map derived from the layout, and a semantic map. To ensure robust multimodal interaction, "Condition Dropout" is employed during training, preventing the model from over-relying on any single input modality.
2.2 Depth Estimation and NeRF Realization
To transition from 2D to 3D, the system performs layout-conditioned depth estimation. The coarse depth from the layout serves as a geometric prior to refine a high-resolution depth map from the generated 360-degree RGB image. Finally, the resulting RGB-D data is used to train a Neural Radiance Field (NeRF), enabling free-viewpoint rendering and immersive exploration of the generated scene.
3. Experimental Results
The proposed method was evaluated across both indoor (Structured3D, Matterport 3D) and outdoor (SceneDreamer, SUN360) domains.
3.1 Quantitative Performance
MaGRITTe demonstrated superior performance compared to state-of-the-art models like PanoDiff. On the Structured3D dataset, MaGRITTe achieved an FID score of 18.87, significantly outperforming PanoDiff's 21.23. This indicates higher visual fidelity and better alignment with the target distribution. Additionally, Intersection over Union (IoU) metrics for object placement confirmed that MaGRITTe adheres more strictly to user-provided layouts than text-only or image-only baselines.
3.2 Qualitative Flexibility
A notable demonstration of the model's flexibility involved using Vermeer’s The Milkmaid as a partial image input. By varying the text prompt and layout, MaGRITTe successfully generated consistent 3D environments ranging from traditional medieval kitchens to modern interior designs, while preserving the identity and style of the original character.
4. Discussion and Limitations
While MaGRITTe represents a significant advancement, several limitations remain:
- Geometric Ambiguity: Overlapping objects in the top-view layout can sometimes lead to merged artifacts in the generated scene.
- Prompt Conflicts: Strong visual cues in the input image may occasionally override conflicting text prompts (e.g., a "modern" prompt vs. a "medieval" image).
- Occlusion Artifacts: While NeRF allows for movement, the quality of regions occluded from the central 360-degree viewpoint is lower than the primary visible areas.
- Resolution Constraints: The current ERP resolution limits the capture of extremely fine 3D details.
5. Conclusion
MaGRITTe provides a powerful tool for democratizing 3D scene creation. By combining the strengths of text-to-image diffusion, layout-based control, and NeRF-based 3D realization, it allows users to generate complex, controllable environments from simple multimodal sketches. Future work will focus on improving resolution and handling complex occlusions more effectively.
Summary updated