RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texture

Liangchen Song, Liangliang Cao, Hongyu Xu, Kai Kang, Feng Tang, Junsong Yuan, Yang Zhao

TL;DR

RoomDreamer is a text-driven 3D synthesis framework that transforms low-quality, real-world indoor scans into highly stylized rooms based on text prompts. By combining geometry-guided 2D diffusion with joint mesh and texture optimization, it simultaneously re-stylizes spaces and fixes scanning artifacts like holes and geometric distortions.

What Problem Does This Paper Address?

Consumer 3D scanning technology (like LiDAR on modern smartphones) has made capturing indoor spaces highly accessible. However, the raw meshes generated by these devices are often plagued by structural defects: holes from missing data, distorted objects, and blurry textures.

While generative AI models excel at creating 3D objects from scratch, applying them to edit or re-stylize an existing real-world room is challenging. Existing methods typically generate views sequentially (frame-by-frame outpainting), which leads to severe visual drift, or they rely purely on image-to-image editing, which restricts creative freedom and fails to correct the underlying broken geometry. RoomDreamer addresses this by treating the scene as a unified whole, optimizing both texture and geometry to match a textual prompt (e.g., "a royal castle") while respecting the original room layout.

Methods

The RoomDreamer pipeline operates in two primary stages:

  1. Geometry Guided Diffusion: To avoid the "striping" artifacts common in sequential view generation, the model first synthesizes a $360^\circ$ central cubemap panorama of the room using a text- and geometry-conditioned 2D diffusion model (ControlNet). Standard depth maps are view-dependent, causing sharp, unnatural seams at the boundaries of cubemap faces. To solve this, the authors introduce a blending scheme that combines standard depth maps (for local structural alignment) with camera-origin distance maps (which are view-independent and ensure smooth boundary transitions). Unseen areas outside the initial cubemap are later filled using masked outpainting.
  2. Mesh Optimization: Using a differentiable renderer, the framework backpropagates gradients to jointly update both vertex colors (texture) and vertex positions (geometry). To clean up the noisy geometry of the initial scan, the authors run an off-the-shelf monocular depth estimator (MiDaS) on the newly generated 2D views to act as pseudo-supervision. A Laplacian loss is applied to ensure that regions identified as flat in 2D remain smooth and planar in 3D, effectively filling holes and straightening warped walls.
Figure 2: The overall framework of RoomDreamer, showing the Geometry Guided Diffusion and Mesh Optimization stages.

Datasets

The authors evaluated RoomDreamer using the ARKitScenes dataset, which consists of diverse, real-world indoor scenes captured via iPhone mobile sensors. They utilized the validation set to demonstrate how the framework handles the messy, imperfect geometry typical of consumer-grade scans.

Key Insights and Ideas

  • The Cubemap Advantage: Generating a full $360^\circ$ view simultaneously anchors the global style of the room, preventing the stylistic drift and boundary mismatches that happen when generating images frame-by-frame.
  • Depth vs. Distance Maps: Standard depth measures distance to the image plane, making it highly sensitive to camera rotation. Distance maps measure absolute distance to the camera origin, making them rotation-invariant. Blending the two allows the diffusion model to maintain sharp local details without creating jarring seams where cubemap faces meet.
Figure 5: Comparison of diffusion using depth maps vs. distance maps at cubemap boundaries.
  • 2D Priors Fix 3D Messes: A powerful generative 2D image model inherently understands what a flat wall or a clean piece of furniture looks like. By extracting monocular depth from these "dreamed" 2D images, the framework can backpropagate structural corrections to patch physical holes and smooth out jagged artifacts in the original 3D mesh.

Key Results

The framework was evaluated quantitatively against prominent 3D generation and editing baselines (Score Distillation Sampling (SDS) and InstructN2N) across 80 meshes and 15 text prompts:

  • Text-Image Similarity: RoomDreamer achieved a CLIP-based similarity score of 0.2543, substantially outperforming SDS (0.1532) and InstructN2N (0.2022), demonstrating a much tighter alignment with user prompts.
  • Direction Consistency: It achieved a consistency score of 0.5281, outperforming the outpainting baseline and proving that the generated styles remain stable across multiple viewing angles.
  • Qualitative Improvements: Visually, RoomDreamer successfully filled large structural gaps in scanned meshes and straightened distorted walls, producing clean, high-fidelity thematic rooms (e.g., transforming a standard living room into a "cozy wooden cabin").

Limitations and Future Work

The authors do not include a dedicated limitations section, but a few constraints are apparent from the text:

  • Dependency on 2D Models: The quality of the final 3D scene is tightly bound to the performance of the underlying 2D diffusion model and the monocular depth estimator. Errors or hallucinations in the 2D generation phase can propagate into the 3D geometry.
  • Computation Time: Optimizing a single scene takes approximately 15 minutes on an NVIDIA A100 GPU, which limits its current viability for real-time or interactive mobile applications.

Summary updated

Abstract

The techniques for 3D indoor scene capturing are widely used, but the meshes produced leave much to be desired. In this paper, we propose "RoomDreamer", which leverages powerful natural language to synthesize a new room with a different style. Unlike existing image synthesis methods, our work addresses the challenge of synthesizing both geometry and texture aligned to the input scene structure and prompt simultaneously. The key insight is that a scene should be treated as a whole, taking into account both scene texture and geometry. The proposed framework consists of two significant components: Geometry Guided Diffusion and Mesh Optimization. Geometry Guided Diffusion for 3D Scene guarantees the consistency of the scene style by applying the 2D prior to the entire scene simultaneously. Mesh Optimization improves the geometry and texture jointly and eliminates the artifacts in the scanned scene. To validate the proposed method, real indoor scenes scanned with smartphones are used for extensive experiments, through which the effectiveness of our method is demonstrated.
BibTeX
@inproceedings{song2023roomdreamer,
  author = {Liangchen Song and Liangliang Cao and Hongyu Xu and Kai Kang and Feng Tang and Junsong Yuan and Yang Zhao},
  title = {{RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texture}},
  booktitle = {ACM Multimedia},
  year = {2023},
}
Figure