Programmable-Room: Interactive Textured 3D Room Meshes Generation Empowered by Large Language Models

Jihyun Kim, Junho Park, Kyeongbo Kong, Suk-Ju Kang

TL;DR

Programmable-Room is a modular framework that uses Large Language Models (LLMs) to translate natural language instructions into executable "visual programs" for 3D room generation. By decomposing the complex task of scene synthesis into discrete steps—like shape determination, panorama texturing, and furniture arrangement—it allows for precise, interactive editing of 3D indoor environments with state-of-the-art visual quality.

What Problem Does This Paper Address?

Creating high-fidelity 3D indoor scenes from text prompts is a "holy grail" in computer vision, but existing methods often fall short in two areas: control and consistency.

Current end-to-end models (like Text2Room or SceneScape) often struggle with global structural coherence or treat the scene as a "black box." If you want to change just the floor texture or move a single chair, you often have to regenerate the entire scene, losing the parts you liked. Furthermore, many models produce "hallucinated" geometry that looks plausible from one angle but falls apart when the camera moves. Programmable-Room addresses this by moving away from monolithic generation toward a modular, "programmable" approach.

Methods

The core innovation is the use of Visual Programming (VP). Instead of the LLM (GPT-4) generating 3D data directly, it acts as an orchestrator. It parses a user's natural language instruction (e.g., "Add a wooden floor and a blue sofa") and writes a Python-like script that calls specific, specialized modules:

  1. Shape & Layout: Modules like GenShape and GenLayout determine the 3D coordinates and boundaries of the room.
  2. PRIG (Panorama Room Image Generation): This is the authors' custom diffusion-based model. Unlike standard text-to-image models, PRIG is conditioned on multiple visual prompts simultaneously: layout maps, depth maps, and semantic maps.
  3. Aha Moment: To ensure the 360-degree panorama is seamless and structurally sound, they use a BiLSTM to create a 1D representation of the room layout. This captures long-range geometric patterns (like the fact that the left edge of a panorama must match the right edge) more efficiently than standard 2D convolutions.
  4. Furniture Generation: It integrates LayoutGPT to arrange furniture in a physically plausible way, outputting a CSS-like file that can be easily edited.
  5. Merging: A final module integrates the textured room mesh with the furniture models.

Datasets

The authors utilized several key benchmarks to train and evaluate the system:

  • Structure3D: Used to train the PRIG model. It contains over 21,000 rooms with rich annotations for panoramas, depth, and semantics.
  • 3D-FUTURE & 3D-FRONT: These provided the industrial-grade 3D CAD furniture shapes and layout patterns used by the furniture modules.
  • Qwen-VL: The authors used this vision-language model to generate descriptive text captions for the Structure3D dataset, creating the text-image pairs needed for diffusion training.

Key Insights and Ideas

  • Surgical Editing: Because the pipeline is modular, the system supports "infinite editing." You can call EditTexture to change the wallpaper without affecting the room's dimensions or the furniture already placed.
  • LLMs as Logic Engines, Not Artists: By using GPT-4 to write code rather than pixels, the system leverages the LLM's strength in reasoning and instruction-following while leaving the "heavy lifting" of 3D rendering to specialized geometric algorithms.
  • The Power of 1D Representation: Representing a 360-degree room as a 1D sequence via BiLSTM is a clever workaround for the "seam" problem in panoramas, ensuring that the generated textures are globally consistent.

Key Results

Programmable-Room outperforms existing state-of-the-art models in both quality and efficiency:

  • Visual Quality: In panorama generation, it achieved a FID of 65.68, significantly lower (better) than competitors like Text2Light (103.39) and PanFusion (72.66).
  • 3D Accuracy: In user studies, it scored highest in Perceptual Quality (3.57/5) and 3D Structure Completeness (3.82/5), beating models like Holodeck and Text2Room.
  • Inference Speed: The system is remarkably fast, generating a scene in roughly 154 seconds, compared to the ~5,179 seconds required by Text2Room.

Limitations and Future Work

While powerful, the authors note a few constraints:

  • Room Variety: The system is currently optimized for residential spaces (bedrooms and living rooms) and may struggle with more complex or unconventional architectural styles.
  • Furniture Dependency: The quality of the final scene is heavily dependent on the library of 3D furniture models available to the GenFurniture module.
  • Future Directions: The team aims to expand the module library to include more diverse room categories and integrate even more realistic, physics-based furniture generation models.

Summary updated

Abstract

We present Programmable-Room, a framework which interactively generates and edits a 3D room mesh, given natural language instructions. For precise control of a room's each attribute, we decompose the challenging task into simpler steps such as creating plausible 3D coordinates for room meshes, generating panorama images for the texture, constructing 3D meshes by integrating the coordinates and panorama texture images, and arranging furniture. To support the various decomposed tasks with a unified framework, we incorporate visual programming (VP). VP is a method that utilizes a large language model (LLM) to write a Python-like program which is an ordered list of necessary modules for the various tasks given in natural language. We develop most of the modules. Especially, for the texture generating module, we utilize a pretrained large-scale diffusion model to generate panorama images conditioned on text and visual prompts (i.e., layout, depth, and semantic map) simultaneously. Specifically, we enhance the panorama image generation quality by optimizing the training objective with a 1D representation of a panorama scene obtained from bidirectional LSTM. We demonstrate Programmable-Room's flexibility in generating and editing 3D room meshes, and prove our framework's superiority to an existing model quantitatively and qualitatively. Project page is available in https://jihyun0510.github.io/Programmable_Room_Page/.
BibTeX
@article{kim2025programmable,
  author = {Jihyun Kim and Junho Park and Kyeongbo Kong and Suk-Ju Kang},
  title = {{Programmable-Room: Interactive Textured 3D Room Meshes Generation Empowered by Large Language Models}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3581748},
}

Loading related content...

Figure