Abstract
We present Programmable-Room, a framework which interactively generates and edits a 3D room mesh, given natural language instructions. For precise control of a room's each attribute, we decompose the challenging task into simpler steps such as creating plausible 3D coordinates for room meshes, generating panorama images for the texture, constructing 3D meshes by integrating the coordinates and panorama texture images, and arranging furniture. To support the various decomposed tasks with a unified framework, we incorporate visual programming (VP). VP is a method that utilizes a large language model (LLM) to write a Python-like program which is an ordered list of necessary modules for the various tasks given in natural language. We develop most of the modules. Especially, for the texture generating module, we utilize a pretrained large-scale diffusion model to generate panorama images conditioned on text and visual prompts (i.e., layout, depth, and semantic map) simultaneously. Specifically, we enhance the panorama image generation quality by optimizing the training objective with a 1D representation of a panorama scene obtained from bidirectional LSTM. We demonstrate Programmable-Room's flexibility in generating and editing 3D room meshes, and prove our framework's superiority to an existing model quantitatively and qualitatively.
Introduction
Creating high-fidelity 3D indoor scenes from text prompts is a "holy grail" in computer vision. However, existing end-to-end models often struggle with two critical areas: control and consistency. Monolithic models frequently treat the scene as a "black box," making it nearly impossible to edit specific elements—like changing a floor texture or moving a single chair—without regenerating the entire environment. Furthermore, many models produce "hallucinated" geometry that lacks global structural coherence.
Programmable-Room addresses these challenges by moving away from monolithic generation toward a modular, "programmable" approach. By decomposing scene synthesis into discrete, manageable steps, the framework allows for precise, interactive editing of 3D indoor environments with state-of-the-art visual quality.
Methodology: Visual Programming for 3D Scenes
The core innovation of Programmable-Room is the use of Visual Programming (VP). Instead of tasking a Large Language Model (LLM) like GPT-4 with direct 3D data generation, the framework utilizes the LLM as an orchestrator.
Modular Decomposition
The LLM parses a user's natural language instruction and writes a Python-like script that calls specific, specialized modules:
- Shape & Layout: Modules like
GenShapeandGenLayoutdetermine the 3D coordinates and boundaries of the room. - Panorama Room Image Generation (PRIG): This custom diffusion-based model is conditioned on multiple visual prompts simultaneously: layout maps, depth maps, and semantic maps. To ensure the 360-degree panorama is seamless, the authors use a BiLSTM to create a 1D representation of the room layout, capturing long-range geometric patterns more efficiently than standard 2D convolutions.
- Furniture Generation: The system integrates
LayoutGPTto arrange industrial-grade 3D CAD furniture models in a physically plausible way. - Merging: A final module integrates the textured room mesh with the furniture models.
Surgical Editing and Logic
Because the pipeline is modular, the system supports "infinite editing." Users can call specific functions like EditTexture to change wallpaper without affecting the room's dimensions or existing furniture. By using GPT-4 to write code rather than pixels, the system leverages the LLM's strength in reasoning and instruction-following while leaving the heavy lifting of 3D rendering to specialized geometric algorithms.
Datasets and Training
The authors utilized several key benchmarks to train and evaluate the system:
- Structure3D: Used to train the PRIG model, containing over 21,000 rooms with rich annotations.
- 3D-FUTURE & 3D-FRONT: Provided the 3D CAD furniture shapes and layout patterns.
- Qwen-VL: Employed to generate descriptive text captions for the Structure3D dataset, creating the necessary text-image pairs for diffusion training.
Key Results
Programmable-Room outperforms existing state-of-the-art models in both quality and efficiency:
- Visual Quality: In panorama generation, it achieved a FID of 65.68, significantly better than competitors like Text2Light (103.39) and PanFusion (72.66).
- 3D Accuracy: In user studies, it scored highest in Perceptual Quality (3.57/5) and 3D Structure Completeness (3.82/5).
- Inference Speed: The system generates a scene in roughly 154 seconds, compared to the ~5,179 seconds required by previous models like Text2Room.
Conclusion
Programmable-Room represents a significant step forward in interactive 3D scene synthesis. While currently optimized for residential spaces, its modular nature provides a foundation for expanding into more complex architectural styles. By combining the logical reasoning of LLMs with specialized visual modules, the framework offers unprecedented control over 3D environment generation.
Summary updated