Introduction
The creation of realistic, interactive 3D indoor environments is essential for fields ranging from embodied AI training to architectural visualization. However, existing automated synthesis methods often struggle with three critical challenges: limited controllability over specific user requirements, a lack of semantic coherence across multi-room layouts, and "functional inertia"—where scenes look plausible but lack the metadata necessary for agent interaction.
RoomPilot addresses these issues by treating scene synthesis as a structured compilation process. It transforms natural language descriptions or CAD floor plans into a unified, hierarchical representation that ensures both physical functionality and design intent.
The RoomPilot Pipeline
The core of the RoomPilot framework is a three-stage pipeline designed to bridge the gap between high-level human intent and low-level geometric realization.
1. Cross-Modal Semantic Parsing
RoomPilot utilizes Large Language Models (LLMs) to extract entities and spatial relationships from text descriptions. Simultaneously, a geometric parser processes CAD floor plans to define structural boundaries. These inputs are compiled into the Indoor Domain-Specific Language (IDSL).
2. The IDSL Hierarchy
The IDSL organizes the scene into a multi-level semantic hierarchy:
- Building Level: Defines the global topology and how rooms connect.
- Room Level: Specifies functional zones (e.g., dining vs. living area) and stylistic constraints.
- Object Level: Details fine-grained attributes, such as furniture dimensions and interaction properties.
3. Self-Regulating Scene Optimization
Rather than a single-pass generation, RoomPilot employs an energy-based evolution process. This process balances Structural Energy (preventing overlaps and wall collisions) with Semantic Energy (ensuring the layout matches the user's functional description). By using "Progressive Rule Activation," the system solves fundamental structural issues before refining aesthetic and functional details.
Interactive Asset Generation
To ensure the generated environments are truly functional, RoomPilot leverages a curated library of over 9,000 interaction-annotated assets. These assets include metadata for canonical poses and semantic tags, allowing AI agents to perform tasks like opening drawers or sitting. For unique requirements, the system integrates 27 category-specific procedural generators to fill gaps where static assets may not suffice.
Key Results and Performance
RoomPilot demonstrates significant improvements over state-of-the-art baselines like Holodeck and LayoutGPT in terms of both controllability and physical plausibility.
- Controllability: In Layout Fidelity (LF) benchmarks, RoomPilot achieved a score of 0.58, more than doubling the performance of LayoutVLM (0.25).
- Constraint Satisfaction: The system reached an 89% success rate in adhering to specific object-count constraints.
- User Evaluation: In subjective studies, RoomPilot was rated 4.42/5 for physical plausibility, effectively eliminating common artifacts such as floating furniture or clipping.
Conclusion
RoomPilot provides a robust framework for synthesizing complex, multi-room 3D environments that are both controllable and interactive. By decoupling design intent from geometric realization through the IDSL, it offers a scalable solution for generating high-fidelity training environments for embodied AI. Future work will focus on extending the framework to multi-floor structures and enhancing real-time dynamic scene interaction.
Summary updated