Text-to-Scene with Large Reasoning Models

Frédéric Berdoz, Luca A. Lanzendörfer, Nick Tuninga, Roger WattenhoferAAAI 2025 · DOI · OpenReview

TL;DR

This paper introduces Reason-3D, a novel text-to-scene generation system that leverages Large Reasoning Models (LRMs) to create 3D environments from natural language descriptions. Reason-3D excels at complex object placement and adherence to intricate instructions, significantly outperforming existing methods in visual fidelity and constraint satisfaction.

What Problem Does This Paper Address?

Creating realistic and coherent 3D scenes from text descriptions is a challenging task. Existing text-to-scene methods often struggle with:

  • Complex Geometries and Object Transformations: Difficulty in accurately placing and orienting objects, especially when rotations or unusual configurations are involved.
  • Adherence to Complex Instructions: Inability to fully capture and execute detailed, multi-part instructions that involve spatial relationships and constraints.
  • Generalization: Limited ability to adapt to new object types, scene settings (like outdoor environments), or instructions outside their training distribution.

These limitations hinder the application of text-to-scene generation in fields like interior design, game development, and robotics, where precise and flexible scene creation is crucial.

Methods

Reason-3D is a modular pipeline that integrates Large Reasoning Models (LRMs) for advanced spatial reasoning. Its core components are:

  1. Object Retrieval:
    • Captioning: An image captioning model generates detailed textual descriptions (physical, functional, contextual attributes) for objects in a library.
  2. Embedding: These descriptions are embedded into vectors and stored in a vector database.
  3. LRM-based Selection: An LRM processes the user's scene prompt to identify required objects. It then uses embedding similarity and its own semantic voting to select the most appropriate object instance from the database, ensuring it aligns with the prompt's intent.
  4. Object Placement:
    • Constraint Extraction: An LRM extracts implicit and explicit spatial constraints from the scene description.
  5. Placement Ordering: Another LRM determines an optimal order for placing objects, ensuring dependencies are met (e.g., placing a table before a cup on it).
  6. Autoregressive Placement: Objects are placed iteratively, with an LRM calculating the position and rotation for each new object based on the scene description, existing objects, and extracted constraints. This process uses a vision-language model (VLM) for initial object orientation.
  7. Collision-Aware Refinement:
    • Collision Detection: After initial placement, the system checks for overlapping bounding boxes.
  8. LRM-based Resolution: The LRM analyzes detected collisions and revises object positions and rotations to resolve them, while also considering semantic validity (e.g., a trash bin under a table might be acceptable). This two-stage approach enhances physical plausibility and reduces reasoning load.

A key novelty is the reliance on off-the-shelf LRMs (like Gemini 2.5 Pro) without task-specific fine-tuning, enabling zero-shot generalization.

Datasets

  • Objaverse dataset: Used as the primary source for 3D objects. A preprocessing step using a VLM is applied to ensure objects are upright and consistently oriented.
  • 3D-FRONT dataset: Mentioned as a dataset used by prior methods for training, but Reason-3D does not require fine-tuning on it.
  • No new benchmarks introduced: The paper evaluates on existing indoor scene synthesis tasks and uses human evaluation studies.

Key Insights and Ideas

  • LRMs as Spatial Reasoners: The paper strongly advocates for the use of LRMs not just for language understanding but for complex spatial reasoning, including object placement, rotation, and collision avoidance.
  • Zero-Shot Generalization: By leveraging the inherent reasoning capabilities of LRMs, Reason-3D achieves impressive generalization to new scenes and instructions without requiring domain-specific training.
  • Modular Pipeline for Robustness: The separation of retrieval, placement, and refinement stages, each powered by LRMs, allows for a robust and flexible system that can handle complex scenarios.
  • Implicit Constraints are Key: The ability to extract and act upon implicit spatial relationships from natural language is crucial for generating realistic scenes.
  • Collision Refinement is Essential: Even with advanced placement, a dedicated refinement step is necessary to ensure physically plausible and collision-free scenes.

Key Results

  • Human Evaluation: Reason-3D significantly outperformed baselines like Holodeck and LayoutVLM in human evaluations for overall scene synthesis quality, visual fidelity, and adherence to constraints. Participants preferred Reason-3D's outputs by a wide margin (e.g., 95.2% win rate against Holodeck, 98.4% against LayoutVLM).
  • Object Retrieval: Reason-3D achieved high top-k accuracies in object retrieval (e.g., 75% top-1, 85% top-5, 90% top-10), outperforming Holodeck.
  • Object Placement: Reason-3D consistently achieved higher scores for object placement quality (e.g., 4.4 on a 1-5 scale) compared to LayoutVLM (e.g., 3.0), especially as instruction complexity increased.
  • Outdoor Scenes: The system demonstrated its ability to generate complex outdoor scenes with many objects, showcasing its generalization capabilities.
  • LRM Benchmarking: Gemini 2.5 Pro was identified as a top-performing LRM for spatial reasoning tasks, outperforming others like GPT-4.1, Claude Sonnet 4, and DeepSeek-R1.

Limitations and Future Work

  • Object Database Alignment: The system assumes objects in the database are pre-aligned upright. Misalignments in other rotational degrees of freedom are not fully handled.
  • Object Scale Consistency: The framework assumes object scales are approximately correct relative to each other.
  • Structural Elements: Reason-3D does not generate structural elements like floors or walls.
  • LRM Dependencies: The system inherits limitations from the underlying LRMs it uses.
  • Future Work: The authors anticipate continued integration and refinement of LRMs for scene synthesis, suggesting that the rapid evolution of these models will lead to further improvements. They also highlight the potential for more complex spatial reasoning tasks.

Summary updated

Abstract

Prompt-driven scene synthesis allows users to generate complete 3D environments from textual descriptions. Current text-to-scene methods often struggle with complex geometries and object transformations, and tend to show weak adherence to complex instructions. We address these limitations by introducing Reason-3D, a text-to-scene model powered by large reasoning models (LRMs). Reason-3D integrates object retrieval using captions covering physical, functional, and contextual attributes. Reason-3D then places the selected objects based on implicit and explicit layout constraints, and refines their positions with collision-aware spatial reasoning. Evaluated on instructions ranging from simple to complex indoor configurations, Reason-3D significantly outperforms previous methods in human-rated visual fidelity, adherence to constraints, and asset retrieval quality. Beyond its contribution to the field of text-to-scene generation, our work showcases the advanced spatial reasoning abilities of modern LRMs. Additionally, we release the codebase to further the research in object retrieval and placement with LRMs.
BibTeX
@inproceedings{berdoz2025text,
  author = {Frédéric Berdoz and Luca A. Lanzendörfer and Nick Tuninga and Roger Wattenhofer},
  title = {{Text-to-Scene with Large Reasoning Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v40i4.37229},
}
Figure