RoomPilot: Controllable Synthesis of Interactive Indoor Environments via Multimodal Semantic Parsing

Wentang Chen, Shougao Zhang, Yiman Zhang, Tianhao Zhou, Ruihui LiarXiv 2025 · DOI · arXiv

Introduction

The creation of realistic, interactive 3D indoor environments is essential for fields ranging from embodied AI training to architectural visualization. However, existing automated synthesis methods often struggle with three critical challenges: limited controllability over specific user requirements, a lack of semantic coherence across multi-room layouts, and "functional inertia"—where scenes look plausible but lack the metadata necessary for agent interaction.

RoomPilot addresses these issues by treating scene synthesis as a structured compilation process. It transforms natural language descriptions or CAD floor plans into a unified, hierarchical representation that ensures both physical functionality and design intent.

The RoomPilot Pipeline

The core of the RoomPilot framework is a three-stage pipeline designed to bridge the gap between high-level human intent and low-level geometric realization.

Overview of the RoomPilot framework, illustrating the transition from multimodal inputs to a hierarchical 3D scene.

1. Cross-Modal Semantic Parsing

RoomPilot utilizes Large Language Models (LLMs) to extract entities and spatial relationships from text descriptions. Simultaneously, a geometric parser processes CAD floor plans to define structural boundaries. These inputs are compiled into the Indoor Domain-Specific Language (IDSL).

2. The IDSL Hierarchy

The IDSL organizes the scene into a multi-level semantic hierarchy:

  • Building Level: Defines the global topology and how rooms connect.
  • Room Level: Specifies functional zones (e.g., dining vs. living area) and stylistic constraints.
  • Object Level: Details fine-grained attributes, such as furniture dimensions and interaction properties.

3. Self-Regulating Scene Optimization

Rather than a single-pass generation, RoomPilot employs an energy-based evolution process. This process balances Structural Energy (preventing overlaps and wall collisions) with Semantic Energy (ensuring the layout matches the user's functional description). By using "Progressive Rule Activation," the system solves fundamental structural issues before refining aesthetic and functional details.

Interactive Asset Generation

To ensure the generated environments are truly functional, RoomPilot leverages a curated library of over 9,000 interaction-annotated assets. These assets include metadata for canonical poses and semantic tags, allowing AI agents to perform tasks like opening drawers or sitting. For unique requirements, the system integrates 27 category-specific procedural generators to fill gaps where static assets may not suffice.

Key Results and Performance

RoomPilot demonstrates significant improvements over state-of-the-art baselines like Holodeck and LayoutGPT in terms of both controllability and physical plausibility.

A generated multi-room apartment demonstrating semantic coherence and functional layout across different zones.
  • Controllability: In Layout Fidelity (LF) benchmarks, RoomPilot achieved a score of 0.58, more than doubling the performance of LayoutVLM (0.25).
  • Constraint Satisfaction: The system reached an 89% success rate in adhering to specific object-count constraints.
  • User Evaluation: In subjective studies, RoomPilot was rated 4.42/5 for physical plausibility, effectively eliminating common artifacts such as floating furniture or clipping.

Conclusion

RoomPilot provides a robust framework for synthesizing complex, multi-room 3D environments that are both controllable and interactive. By decoupling design intent from geometric realization through the IDSL, it offers a scalable solution for generating high-fidelity training environments for embodied AI. Future work will focus on extending the framework to multi-floor structures and enhancing real-time dynamic scene interaction.

Summary updated

Abstract

Generating controllable indoor scenes is fundamental to applications in game development, architectural visualization, and embodied AI. However, existing approaches either support a limited input modalities or rely on implicit generation processes that hinder precise control over scene structure and semantics. To address these limitations, we introduce RoomPilot, a unified framework for controllable indoor scene synthesis from multi-modal inputs, including textual descriptions and CAD floor plans. RoomPilot maps heterogeneous inputs into an Indoor Domain-Specific Language (IDSL), which serves as a structured and interpretable semantic representation for describing indoor scenes. Built upon IDSL, RoomPilot presents a hierarchical synthesis pipeline that progressively organizes scenes at the building, room, and object levels, promoting structural coherence and functional consistency across multi-room layouts. Moreover, RoomPilot constructs a curated asset dataset with rich semantic annotations to support high-quality scene synthesis, improving visual realism and appearance consistency. Extensive experiments demonstrate effective multi-modal understanding, fine-grained controllability in scene generation, and improved physical consistency and visual fidelity, marking a significant step toward controllable 3D indoor scene synthesis. Code and model will be available.
BibTeX
@misc{chen2025roompilot,
  author = {Wentang Chen and Shougao Zhang and Yiman Zhang and Tianhao Zhou and Ruihui Li},
  title = {{RoomPilot: Controllable Synthesis of Interactive Indoor Environments via Multimodal Semantic Parsing}},
  year = {2025},
  eprint = {2512.11234},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.11234},
}
Figure