TL;DR
Programmable-Room is a modular framework that uses Large Language Models (LLMs) to translate natural language instructions into executable "visual programs" for 3D room generation. By decomposing the complex task of scene synthesis into discrete steps—like shape determination, panorama texturing, and furniture arrangement—it allows for precise, interactive editing of 3D indoor environments with state-of-the-art visual quality.
What Problem Does This Paper Address?
Creating high-fidelity 3D indoor scenes from text prompts is a "holy grail" in computer vision, but existing methods often fall short in two areas: control and consistency.
Current end-to-end models (like Text2Room or SceneScape) often struggle with global structural coherence or treat the scene as a "black box." If you want to change just the floor texture or move a single chair, you often have to regenerate the entire scene, losing the parts you liked. Furthermore, many models produce "hallucinated" geometry that looks plausible from one angle but falls apart when the camera moves. Programmable-Room addresses this by moving away from monolithic generation toward a modular, "programmable" approach.
Methods
The core innovation is the use of Visual Programming (VP). Instead of the LLM (GPT-4) generating 3D data directly, it acts as an orchestrator. It parses a user's natural language instruction (e.g., "Add a wooden floor and a blue sofa") and writes a Python-like script that calls specific, specialized modules:
- Shape & Layout: Modules like
GenShapeandGenLayoutdetermine the 3D coordinates and boundaries of the room. - PRIG (Panorama Room Image Generation): This is the authors' custom diffusion-based model. Unlike standard text-to-image models, PRIG is conditioned on multiple visual prompts simultaneously: layout maps, depth maps, and semantic maps.
- Aha Moment: To ensure the 360-degree panorama is seamless and structurally sound, they use a BiLSTM to create a 1D representation of the room layout. This captures long-range geometric patterns (like the fact that the left edge of a panorama must match the right edge) more efficiently than standard 2D convolutions.
- Furniture Generation: It integrates
LayoutGPTto arrange furniture in a physically plausible way, outputting a CSS-like file that can be easily edited. - Merging: A final module integrates the textured room mesh with the furniture models.
Datasets
The authors utilized several key benchmarks to train and evaluate the system:
- Structure3D: Used to train the PRIG model. It contains over 21,000 rooms with rich annotations for panoramas, depth, and semantics.
- 3D-FUTURE & 3D-FRONT: These provided the industrial-grade 3D CAD furniture shapes and layout patterns used by the furniture modules.
- Qwen-VL: The authors used this vision-language model to generate descriptive text captions for the Structure3D dataset, creating the text-image pairs needed for diffusion training.
Key Insights and Ideas
- Surgical Editing: Because the pipeline is modular, the system supports "infinite editing." You can call
EditTextureto change the wallpaper without affecting the room's dimensions or the furniture already placed. - LLMs as Logic Engines, Not Artists: By using GPT-4 to write code rather than pixels, the system leverages the LLM's strength in reasoning and instruction-following while leaving the "heavy lifting" of 3D rendering to specialized geometric algorithms.
- The Power of 1D Representation: Representing a 360-degree room as a 1D sequence via BiLSTM is a clever workaround for the "seam" problem in panoramas, ensuring that the generated textures are globally consistent.
Key Results
Programmable-Room outperforms existing state-of-the-art models in both quality and efficiency:
- Visual Quality: In panorama generation, it achieved a FID of 65.68, significantly lower (better) than competitors like Text2Light (103.39) and PanFusion (72.66).
- 3D Accuracy: In user studies, it scored highest in Perceptual Quality (3.57/5) and 3D Structure Completeness (3.82/5), beating models like Holodeck and Text2Room.
- Inference Speed: The system is remarkably fast, generating a scene in roughly 154 seconds, compared to the ~5,179 seconds required by Text2Room.
Limitations and Future Work
While powerful, the authors note a few constraints:
- Room Variety: The system is currently optimized for residential spaces (bedrooms and living rooms) and may struggle with more complex or unconventional architectural styles.
- Furniture Dependency: The quality of the final scene is heavily dependent on the library of 3D furniture models available to the
GenFurnituremodule. - Future Directions: The team aims to expand the module library to include more diverse room categories and integrate even more realistic, physics-based furniture generation models.
Summary updated