MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Text

Takayuki Hara, Tatsuya Harada

Creating a 3D scene usually feels like a choice between two extremes: you either spend hours meticulously placing assets in a game engine, or you roll the dice with a generative AI that might ignore half your instructions. MaGRITTe (Manipulative and Generative 3D Realization from Image, Topview and Text) offers a middle ground, allowing users to "conduct" the creation of a 3D environment using a partial image, a top-view layout, and a text prompt.

TL;DR

MaGRITTe is a framework that generates controllable 3D scenes by combining partial images, top-view layouts, and text prompts. By using 360-degree equirectangular images as a common bridge and fine-tuning diffusion models on small synthetic datasets, it achieves high-quality 3D realization without the need for massive, hard-to-acquire 3D training data.

What Problem Does This Paper Address?

Generating 3D scenes that are both high-quality and highly controllable is a "holy grail" in computer vision. Current methods usually struggle with one of three things:

  1. Input Limitations: Most models take only one type of input (e.g., just text or just a single image). Text is great for "vibes" but bad for precise object placement; layouts are great for placement but lack visual style.
  2. Data Scarcity: Training a model to understand 3D geometry typically requires massive datasets of 3D scans, which are expensive and difficult to collect compared to 2D images.
  3. Domain Rigidity: Many models are "locked" into specific environments (like just indoor rooms) because their layout representations don't translate well to other domains like outdoor landscapes.

MaGRITTe tackles these by decomposing the problem: first, it generates a consistent 2D 360-degree panorama from multimodal inputs, and then it "lifts" that 2D data into a 3D Neural Radiance Field (NeRF).

Methods

The MaGRITTe pipeline consists of four main stages:

  1. ERP Conversion: All spatial inputs (the partial image and the top-view layout) are converted into Equirectangular Projection (ERP) format. This is the standard "unwrapped globe" view used for 360-degree photos. This puts everything—from a floor plan to a camera snap—into a single coordinate system.
  2. 360-Degree RGB Generation: The authors fine-tune a pre-trained Stable Diffusion model using ControlNet. They feed it the ERP-formatted partial image, a coarse depth map derived from the layout, and a semantic map. To ensure the model doesn't "forget" how to handle text or become too reliant on one input, they use Condition Dropout, randomly hiding some inputs during training.
  3. Layout-Conditioned Depth Estimation: To turn the 2D panorama into 3D, the model needs an accurate depth map. MaGRITTe uses the coarse layout depth as a guide to refine a high-resolution depth map from the generated RGB image.
  4. NeRF Training: Finally, the 360-degree RGB-D data is used to train a NeRF. This allows users to "walk through" the scene and render it from any perspective, not just the central viewpoint.

Datasets

The researchers used a mix of synthetic "base" datasets and real-world "auxiliary" datasets:

  • Indoor: Modified Structured 3D (synthetic rooms with floor plans) and Matterport 3D (real-world indoor scans).
  • Outdoor: SceneDreamer (synthetic landscapes generated from terrain maps) and SUN360 (real-world outdoor panoramas).
  • Custom Annotations: They used BLIP to automatically generate text captions for these datasets to facilitate multimodal training.

Key Insights and Ideas

  • ERP as the Universal Language: By converting everything to ERP format, the model can reason about the interaction between a "table" in a floor plan and a "window" in a partial image within the same latent space.
  • Synthetic to Real Transfer: The authors proved that you don't need a massive 3D dataset. Fine-tuning a powerful 2D model (Stable Diffusion) on a relatively small amount of synthetic layout data allows it to generalize to real-world "in-the-wild" user sketches.
  • Complementary Modalities: The paper highlights how inputs compensate for each other: the image provides the "what" (style/texture), the layout provides the "where" (geometry), and the text provides the "context" (overall atmosphere).

Key Results

  • Quantitative: MaGRITTe outperformed PanoDiff (a state-of-the-art 360-gen model) on the Structured3D dataset, achieving a significantly better FID score (18.87 vs 21.23), indicating higher image quality and realism.
  • Qualitative: The model demonstrated impressive "creative" flexibility. In one example, it took Vermeer’s The Milkmaid as a partial image and, depending on the text prompt and layout, successfully transformed the scene into either a medieval kitchen or a sleek, modern one while maintaining the original character's appearance.
  • Controllability: Evaluation of object placement (via IoU) showed that MaGRITTe is much better at respecting the user's layout than previous text-and-image-only methods.

Limitations and Future Work

  • Layout Overlap: If objects in the top-view layout overlap from the observer's perspective, the model sometimes merges them into a single blob.
  • Prompt Conflict: Occasionally, the model ignores text prompts if they strongly conflict with the provided image or layout (e.g., asking for a "modern" style when the input image is clearly "medieval").
  • Occlusion Quality: While the NeRF allows for free movement, the image quality drops significantly when looking at the "back" of objects that were hidden from the central 360-degree view.
  • Resolution: The resolution of ERP images is currently a bottleneck for fine 3D details.

MaGRITTe represents a significant step toward "democratizing" 3D scene creation, moving us closer to a world where anyone can sketch a floor plan and describe a dream to see it realized in three dimensions.

Summary updated

Abstract

The generation of 3D scenes from user-specified conditions offers a promising avenue for alleviating the production burden in 3D applications. Previous studies required significant effort to realize the desired scene, owing to limited control conditions. We propose a method for controlling and generating 3D scenes under multimodal conditions using partial images, layout information represented in the top view, and text prompts. Combining these conditions to generate a 3D scene involves the following significant difficulties: (1) the creation of large datasets, (2) reflection on the interaction of multimodal conditions, and (3) domain dependence of the layout conditions. We decompose the process of 3D scene generation into 2D image generation from the given conditions and 3D scene generation from 2D images. 2D image generation is achieved by fine-tuning a pretrained text-to-image model with a small artificial dataset of partial images and layouts, and 3D scene generation is achieved by layout-conditioned depth estimation and neural radiance fields (NeRF), thereby avoiding the creation of large datasets. The use of a common representation of spatial information using 360-degree images allows for the consideration of multimodal condition interactions and reduces the domain dependence of the layout control. The experimental results qualitatively and quantitatively demonstrated that the proposed method can generate 3D scenes in diverse domains, from indoor to outdoor, according to multimodal conditions.
BibTeX
@misc{hara2024magritte,
  author = {Takayuki Hara and Tatsuya Harada},
  title = {{MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Text}},
  year = {2024},
  eprint = {2404.00345},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.00345},
}

Loading related content...

Figure