ReSpace: Text-Driven 3D Scene Synthesis and Editing with Preference Alignment

Martin J. J. Bucher, Iro ArmeniarXiv 2025 · arXiv · OpenReview

Abstract

Scene synthesis and editing has emerged as a promising direction in computer graphics. Current trained approaches for 3D indoor scenes either oversimplify object semantics through one-hot class encodings, require masked diffusion for editing, ignore room boundaries, or rely on floor plan renderings that fail to capture complex layouts. In contrast, LLM-based methods enable richer semantics via natural language but do not support editing, remain limited to rectangular layouts, or rely on weak spatial reasoning. We introduce ReSpace, a generative framework for text-driven 3D indoor scene synthesis and editing using autoregressive language models. Our approach features a compact structured scene representation with explicit room boundaries that frames scene editing as a next-token prediction task. We leverage a dual-stage training approach combining supervised fine-tuning and preference alignment, enabling a specially trained language model for object addition that accounts for user instructions, spatial geometry, object semantics, and scene-level composition. For scene editing, we employ a zero-shot LLM to handle object removal and prompts for addition. We further introduce a novel voxelization-based evaluation that captures fine-grained geometry beyond 3D bounding boxes. Experimental results surpass state-of-the-art on object addition while maintaining competitive results on full scene synthesis.

Introduction

Designing realistic 3D indoor scenes requires a delicate balance of aesthetics, semantics, and geometry. Traditional computer vision models often treat this as a coordinate-prediction problem, oversimplifying objects into generic categories like "chair" or "table." This leads to a "semantic gap" where rich details—such as a "modern tufted dark gray sofa"—are lost. Furthermore, existing methods often struggle with non-rectangular room boundaries or lack intuitive editing capabilities, making it difficult for users to modify scenes through simple text commands.

ReSpace addresses these challenges by framing 3D scene synthesis and editing as a next-token prediction task. By representing 3D space as a Structured Scene Representation (SSR) in JSON format, ReSpace leverages the reasoning power of Large Language Models (LLMs) to build and modify rooms. This approach allows for intuitive, text-driven editing while maintaining high geometric precision.

Iterative scene editing workflow in ReSpace, demonstrating object addition, removal, and replacement via natural language prompts.

Methodology

The ReSpace pipeline is built upon three core components designed to bridge the gap between natural language and 3D geometry.

Structured Scene Representation (SSR)

The SSR is a lightweight, text-based format that describes a room's type, its polygonal boundaries, and a list of objects. Each object entry includes textual descriptions, dimensions, positions, and rotations. This structured format allows an LLM to "read" the current state of a scene and "write" new objects into it.

SG-LLM and Dual-Stage Training

The Scene Graph LLM (SG-LLM) is a specialized 1.5B parameter model (based on Qwen2.5) trained for object addition. The training process involves two stages:

  1. Supervised Fine-Tuning (SFT): The model is trained on thousands of existing 3D scenes converted to the SSR format.
  2. Preference Alignment (GRPO): Using Group Relative Policy Optimization, the model is rewarded for generating valid JSON and avoiding layout violations, such as collisions or out-of-bounds placements.

Voxelization-Based Loss (VBL)

To improve geometric accuracy, ReSpace introduces Voxelization-Based Loss (VBL). Unlike standard metrics that use 3D bounding boxes, VBL voxelizes actual 3D meshes. This allows the model to learn complex spatial relationships, such as a chair sliding partially under a table, while strictly penalizing clipping through solid walls.

The ReSpace system architecture, illustrating the flow from user prompt and initial scene to structured representation, LLM processing, and final asset sampling.

Datasets and Evaluation

The researchers developed SSR-3DFRONT, a dataset derived from 3D-FRONT and 3D-FUTURE. They utilized GPT-4o to generate detailed textual descriptions for thousands of 3D assets based on visual renderings, resulting in approximately 13,000 valid indoor scenes in SSR format.

Evaluation focused on geometric accuracy and human preference. ReSpace significantly reduced Out-of-Bounds (OOB) violations compared to baselines like ATISS and MiDiffusion. In the "all" scene category, ReSpace achieved a VBL score of 21.78, a substantial improvement over ATISS's 136.14. In human preference studies involving 25,000 pairwise comparisons, ReSpace achieved a win rate of 50.3% - 54.2%, outperforming state-of-the-art baselines which typically scored between 38% and 41%.

Conclusion

ReSpace demonstrates that treating 3D scene design as a language modeling task enables more intuitive and semantically rich synthesis and editing. By applying preference alignment to geometric constraints, the model achieves superior physical realism.

Future work may address current limitations, such as the "greedy" nature of autoregressive placement, which can lead to suboptimal layouts if early objects are poorly positioned. The authors suggest exploring Monte Carlo Tree Search (MCTS) to optimize placement sequences. Additionally, expanding the framework to handle multi-room floor plans and architectural features like windows and doors remains an open area for research.

Summary updated

Abstract

Scene synthesis and editing has emerged as a promising direction in computer graphics. Current trained approaches for 3D indoor scenes either oversimplify object semantics through one-hot class encodings (e.g., 'chair' or 'table'), require masked diffusion for editing, ignore room boundaries, or rely on floor plan renderings that fail to capture complex layouts. In contrast, LLM-based methods enable richer semantics via natural language (e.g., 'modern studio with light wood furniture') but do not support editing, remain limited to rectangular layouts or rely on weak spatial reasoning from implicit world models. We introduce ReSpace, a generative framework for text-driven 3D indoor scene synthesis and editing using autoregressive language models. Our approach features a compact structured scene representation with explicit room boundaries that frames scene editing as a next-token prediction task. We leverage a dual-stage training approach combining supervised fine-tuning and preference alignment, enabling a specially trained language model for object addition that accounts for user instructions, spatial geometry, object semantics, and scene-level composition. For scene editing, we employ a zero-shot LLM to handle object removal and prompts for addition. We further introduce a novel voxelization-based evaluation that captures fine-grained geometry beyond 3D bounding boxes. Experimental results surpass state-of-the-art on object addition while maintaining competitive results on full scene synthesis.
BibTeX
@misc{bucher2025respace,
  author = {Martin J. J. Bucher and Iro Armeni},
  title = {{ReSpace: Text-Driven 3D Scene Synthesis and Editing with Preference Alignment}},
  year = {2025},
  eprint = {2506.02459},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.02459},
}
Figure