FreeInsert: Disentangled Text-Guided Object Insertion in 3D Gaussian Scene without Spatial Priors

Chenxi Li, Weijie Wang, Qiang Li, Nicu Sebe, Bruno Lepri, Weizhi Nie

Abstract

Editing 3D scenes through object insertion has traditionally required precise manual spatial priors, such as 2D masks or 3D bounding boxes. FreeInsert introduces a novel framework that eliminates these requirements, allowing for semantically precise and visually consistent object insertion in 3D Gaussian Splatting (3DGS) scenes using only high-level text or image prompts. By disentangling object generation from spatial placement and leveraging the reasoning capabilities of Multimodal Large Language Models (MLLMs), FreeInsert achieves state-of-the-art performance in both spatial accuracy and visual integration.

Introduction

While 3D scene editing has advanced rapidly, the task of object insertion remains a significant hurdle. Current state-of-the-art methods typically rely on "spatial priors"—manual inputs that tell the model exactly where to place a new object. Without these, models often struggle with spatial reasoning (e.g., placing a mustache on a forehead), 3D inconsistency (objects that "float" or change appearance across views), and poor environmental integration. FreeInsert aims to make 3D editing as intuitive as a conversation, allowing users to simply provide a prompt like "Add a bowtie to the doll" and letting the system handle the "what," "where," and "how."

Methodology

The FreeInsert pipeline is structured into four distinct stages that disentangle the generation of the object from its placement within the scene.

1. MLLM-based Object Insertion Parser (OIP)

The process begins with an MLLM (such as GPT-4 or Molmo) parsing the user's instruction into structured semantics. This includes identifying the object type, the target attachment region (e.g., "neck"), and the intended spatial relationship (e.g., "align with").

2. Initialization via Large Models

To ensure 3D consistency, the object is first generated as a standalone 3D Gaussian model. A text-to-image model creates a reference image, which is then reconstructed into 3D using a Large Reconstruction Model (LGM). Simultaneously, the MLLM predicts initial scale and rotation, while an open-vocabulary detector (Florence-2) identifies the 2D attachment area in the scene to initialize the 3D position.

3. Hierarchical Spatial-Aware Refinement

The object’s pose is optimized using Spatially-aware Score Distillation Sampling (SSDS). This stage employs a hierarchical loss that balances global scene semantics with local positional constraints. This allows the object to "settle" precisely into the correct location based on the scene's geometry.

4. Object Appearance Refinement

Finally, the system fine-tunes the object's appearance using Low-Rank Adaptation (LoRA) and multi-view sampling. This ensures high visual fidelity and seamless integration with the existing scene's lighting, shadows, and overall style.

Experimental Results

The researchers evaluated FreeInsert across object-centric, human-centric, and complex outdoor environments. Quantitative results demonstrate that FreeInsert significantly outperforms existing baselines and direct MLLM predictions.

Example of a blue bowtie inserted onto a plush doll using FreeInsert, demonstrating precise spatial alignment and visual integration.

In a user study with 50 participants, FreeInsert received a User Vote score of 36.9, outperforming the next best method, TIP-Editor (32.3), which requires manual bounding boxes. Furthermore, in terms of mean Intersection over Union (mIoU) for object placement, FreeInsert achieved 89.5%, vastly exceeding direct predictions from GPT-4V (68.2%) and Molmo-7B (74.7%).

A horse statue inserted into a complex outdoor 3D scene, showcasing the framework's ability to handle diverse environments without manual priors.

Discussion and Insights

A key insight of this work is the use of MLLMs as "spatial architects." MLLMs possess human commonsense regarding spatial relationships that standard diffusion models often lack. By using them to guide initial placement, FreeInsert avoids the "hallucinated" or nonsensical placements common in purely generative approaches. Additionally, the disentanglement of generation and placement prevents the multi-view inconsistency issues that typically occur when trying to "lift" 2D edits into 3D space.

Conclusion and Limitations

FreeInsert represents a major step toward unsupervised, intuitive 3D scene editing. While powerful, the system is still subject to occasional MLLM failures in interpreting highly complex spatial relations. Future work will focus on more complex physical interactions, such as objects being held or partially obscured by scene elements, and further enhancing spatial reasoning for open-world environments.

Summary updated

BibTeX
@inproceedings{li2025freeinsert,
  author = {Chenxi Li and Weijie Wang and Qiang Li and Nicu Sebe and Bruno Lepri and Weizhi Nie},
  title = {{FreeInsert: Disentangled Text-Guided Object Insertion in 3D Gaussian Scene without Spatial Priors}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3755072},
}
Figure