TL;DR
RoomDreamer is a text-driven 3D synthesis framework that transforms low-quality, real-world indoor scans into highly stylized rooms based on text prompts. By combining geometry-guided 2D diffusion with joint mesh and texture optimization, it simultaneously re-stylizes spaces and fixes scanning artifacts like holes and geometric distortions.
What Problem Does This Paper Address?
Consumer 3D scanning technology (like LiDAR on modern smartphones) has made capturing indoor spaces highly accessible. However, the raw meshes generated by these devices are often plagued by structural defects: holes from missing data, distorted objects, and blurry textures.
While generative AI models excel at creating 3D objects from scratch, applying them to edit or re-stylize an existing real-world room is challenging. Existing methods typically generate views sequentially (frame-by-frame outpainting), which leads to severe visual drift, or they rely purely on image-to-image editing, which restricts creative freedom and fails to correct the underlying broken geometry. RoomDreamer addresses this by treating the scene as a unified whole, optimizing both texture and geometry to match a textual prompt (e.g., "a royal castle") while respecting the original room layout.
Methods
The RoomDreamer pipeline operates in two primary stages:
- Geometry Guided Diffusion: To avoid the "striping" artifacts common in sequential view generation, the model first synthesizes a $360^\circ$ central cubemap panorama of the room using a text- and geometry-conditioned 2D diffusion model (ControlNet). Standard depth maps are view-dependent, causing sharp, unnatural seams at the boundaries of cubemap faces. To solve this, the authors introduce a blending scheme that combines standard depth maps (for local structural alignment) with camera-origin distance maps (which are view-independent and ensure smooth boundary transitions). Unseen areas outside the initial cubemap are later filled using masked outpainting.
- Mesh Optimization: Using a differentiable renderer, the framework backpropagates gradients to jointly update both vertex colors (texture) and vertex positions (geometry). To clean up the noisy geometry of the initial scan, the authors run an off-the-shelf monocular depth estimator (MiDaS) on the newly generated 2D views to act as pseudo-supervision. A Laplacian loss is applied to ensure that regions identified as flat in 2D remain smooth and planar in 3D, effectively filling holes and straightening warped walls.
Datasets
The authors evaluated RoomDreamer using the ARKitScenes dataset, which consists of diverse, real-world indoor scenes captured via iPhone mobile sensors. They utilized the validation set to demonstrate how the framework handles the messy, imperfect geometry typical of consumer-grade scans.
Key Insights and Ideas
- The Cubemap Advantage: Generating a full $360^\circ$ view simultaneously anchors the global style of the room, preventing the stylistic drift and boundary mismatches that happen when generating images frame-by-frame.
- Depth vs. Distance Maps: Standard depth measures distance to the image plane, making it highly sensitive to camera rotation. Distance maps measure absolute distance to the camera origin, making them rotation-invariant. Blending the two allows the diffusion model to maintain sharp local details without creating jarring seams where cubemap faces meet.
- 2D Priors Fix 3D Messes: A powerful generative 2D image model inherently understands what a flat wall or a clean piece of furniture looks like. By extracting monocular depth from these "dreamed" 2D images, the framework can backpropagate structural corrections to patch physical holes and smooth out jagged artifacts in the original 3D mesh.
Key Results
The framework was evaluated quantitatively against prominent 3D generation and editing baselines (Score Distillation Sampling (SDS) and InstructN2N) across 80 meshes and 15 text prompts:
- Text-Image Similarity: RoomDreamer achieved a CLIP-based similarity score of 0.2543, substantially outperforming SDS (0.1532) and InstructN2N (0.2022), demonstrating a much tighter alignment with user prompts.
- Direction Consistency: It achieved a consistency score of 0.5281, outperforming the outpainting baseline and proving that the generated styles remain stable across multiple viewing angles.
- Qualitative Improvements: Visually, RoomDreamer successfully filled large structural gaps in scanned meshes and straightened distorted walls, producing clean, high-fidelity thematic rooms (e.g., transforming a standard living room into a "cozy wooden cabin").
Limitations and Future Work
The authors do not include a dedicated limitations section, but a few constraints are apparent from the text:
- Dependency on 2D Models: The quality of the final 3D scene is tightly bound to the performance of the underlying 2D diffusion model and the monocular depth estimator. Errors or hallucinations in the 2D generation phase can propagate into the 3D geometry.
- Computation Time: Optimizing a single scene takes approximately 15 minutes on an NVIDIA A100 GPU, which limits its current viability for real-time or interactive mobile applications.
Summary updated