Global-Local Tree Search in VLMs for 3D Indoor Scene Generation

Wei Deng, Mengshi Qi, Huadong Ma

Abstract

Generating realistic 3D indoor environments from text prompts remains a significant challenge for Vision-Language Models (VLMs). Traditional autoregressive approaches often suffer from error accumulation, where a single misplaced object leads to a cascade of spatial inconsistencies. We present a novel global-local tree search framework that treats scene generation as a hierarchical planning problem. By enabling the model to backtrack and refine its decisions through Depth-First Search (DFS) and utilizing a unique emoji-based grid representation for spatial grounding, our method significantly improves the spatial common sense and realism of generated 3D scenes. In human user studies, our approach achieved a Mean Reciprocal Rank (MRR) of 0.793, substantially outperforming existing state-of-the-art baselines.

Introduction

Creating a 3D indoor scene from a simple text prompt, such as "a cozy bedroom with a reading nook," requires more than just linguistic understanding; it requires spatial common sense. While current VLMs like GPT-4o are adept at following instructions, they often struggle with the geometric constraints of 3D space.

Existing methodologies typically rely on a linear "chain-of-thought" reasoning process. In these models, objects are placed sequentially. However, if an early placement is suboptimal—such as a bed blocking a doorway—the model lacks the mechanism to "undo" its choice. This leads to "hallucinated" layouts characterized by floating furniture, overlapping objects, or items placed outside the room boundaries. To address this, we propose moving from linear generation to active planning via a global-local tree search.

Methodology

Hierarchical Scene Representation

To bridge the gap between high-level text and low-level coordinates, we implement a hierarchical structure: Room → Region → Floor Object → Supported Object. This hierarchy ensures that objects are placed in a logical order; for example, a toaster is only considered after a countertop (the supported object's parent) has been placed.

The core of our framework is the Global Tree Search, which treats the scene layout as a search tree where each node represents an object placement. If the VLM encounters a constraint violation (e.g., running out of floor space), it utilizes Depth-First Search (DFS) to backtrack to a previous state, adjust the prior placement, and attempt a new path.

Complementing this is the Local Tree Search, which handles the micro-positioning of individual objects. The model identifies an "anchor" object within a region and performs a localized search to determine the optimal side, row, and column for the new object relative to that anchor.

Emoji Grid Prompting

VLMs often struggle with raw numerical coordinates. To mitigate this, we discretized the 2D floor plan into a grid where each cell is represented by a unique emoji. This transforms the spatial coordinate problem into a token-selection task. The VLM "places" an object by identifying the specific emoji tokens it should occupy, leveraging the model's strength in pattern recognition and token association rather than precise mathematical calculation.

Experimental Setup

Datasets and Implementation

We utilized Objaverse-1.0 as our primary library for retrieving 3D assets based on the VLM's generated descriptions. To evaluate the system, we developed a custom prompt set consisting of 120 diverse scenarios across four categories: bathrooms, bedrooms, kitchens, and living rooms.

Metrics

Performance was measured using two primary metrics:

  1. CLIP Scores: To evaluate the semantic alignment between the generated 3D scene and the original text prompt.
  2. Human User Study (Reciprocal Rank): To assess the subjective realism and spatial logic of the layouts.

Results and Discussion

Quantitative Performance

Our global-local tree search achieved a Mean Reciprocal Rank (MRR) of 0.793, significantly higher than previous state-of-the-art models like HoloDeck (0.596) and AnyHome (0.443). Ablation studies confirmed that the tree search mechanism is superior to standard Input-Output (IO) and Chain-of-Thought (CoT) prompting, particularly in reducing object collisions and boundary violations.

Qualitative Insights

The ability to backtrack proved essential for solving 3D generation as a constraint-satisfaction problem. Qualitatively, our model demonstrated superior alignment—such as centering a TV stand perfectly in front of a sofa—whereas baseline models often placed objects randomly along walls without regard for functional relationships.

Limitations

Despite its success, the method has limitations. The iterative nature of tree search requires multiple VLM API calls, making it more computationally expensive and slower than single-pass models. Additionally, the model showed slightly lower performance in kitchen environments, where layouts are often governed by rigid, wall-aligned cabinetry that does not always fit the "furniture cluster" logic used in our search algorithm.

Conclusion

By shifting the paradigm from sequential generation to hierarchical planning with backtracking, we have demonstrated a more robust way for VLMs to navigate the complexities of 3D spatial reasoning. Future work will focus on extending this logic to outdoor environments and optimizing the search process for real-time applications in AR/VR.

Summary updated

Abstract

Large Vision-Language Models (VLMs), such as GPT-4, have achieved remarkable success across various fields. However, there are few studies on 3D indoor scene generation with VLMs. This paper considers this task as a planning problem subject to spatial and layout common sense constraints. To solve the problem with a VLM, we propose a new global-local tree search algorithm. Globally, the method places each object sequentially and explores multiple placements during each placement process, where the problem space is represented as a tree. To reduce the depth of the tree, we decompose the scene structure hierarchically, i.e. room level, region level, floor object level, and supported object level. The algorithm independently generates the floor objects in different regions and supported objects placed on different floor objects. Locally, we also decompose the sub-task, the placement of each object, into multiple steps. The algorithm searches the tree of problem space. To leverage the VLM model to produce positions of objects, we discretize the top-down view space as a dense grid and fill each cell with diverse emojis to make to cells distinct. We prompt the VLM with the emoji grid and the VLM produces a reasonable location for the object by describing the position with the name of emojis. The quantitative and qualitative experimental results illustrate our approach generates more plausible 3D scenes than state-of-the-art approaches. Our source code is available at https://github.com/dw-dengwei/TreeSearchGen .
BibTeX
@inproceedings{deng2025global,
  author = {Wei Deng and Mengshi Qi and Huadong Ma},
  title = {{Global-Local Tree Search in VLMs for 3D Indoor Scene Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00839},
}
Figure