GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions

Junjie Wang, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, Qi Tian

Abstract

Recently, impressive results have been achieved in 3D scene editing with text instructions based on a 2D diffusion model. However, current diffusion models primarily generate images by predicting noise in the latent space, and the editing is usually applied to the whole image, which makes it challenging to perform delicate, especially localized, editing for 3D scenes. Inspired by recent 3D Gaussian splatting, we propose a systematic framework, named GaussianEditor, to edit 3D scenes delicately via 3D Gaussians with text instructions. Benefiting from the explicit property of 3D Gaussians, we design a series of techniques to achieve delicate editing. Specifically, we first extract the region of interest (RoI) corresponding to the text instruction, aligning it to 3D Gaussians. The Gaussian RoI is further used to control the editing process. Our framework can achieve more delicate and precise editing of 3D scenes than previous methods while enjoying much faster training speed, i.e. within 20 minutes on a single V100 GPU, more than twice as fast as Instruct-NeRF2NeRF (45 minutes - 2 hours). The project page is at GaussianEditor.github.io.

Introduction

Editing 3D scenes based on natural language instructions (e.g., "make the road look like a river") has traditionally relied on Neural Radiance Fields (NeRFs). However, NeRFs are implicit representations; the entire scene is encoded within the weights of a neural network. This spatial "coupling" makes it incredibly difficult to perform localized edits without accidentally altering the background or unrelated objects.

Furthermore, existing diffusion-based editing tools often apply changes globally. If a user requests to "give the man a red nose," a standard model might redden his entire face or the background. Finally, NeRF-based editing is computationally expensive, often requiring hours of optimization for a single scene. GaussianEditor addresses these issues by leveraging the explicit nature of 3D Gaussian Splatting to provide "surgical" precision in 3D editing.

Methodology

GaussianEditor shifts the backbone from NeRFs to 3D Gaussian Splatting (3D-GS). Because 3D-GS represents a scene as millions of individual, explicit "blobs" (Gaussians), it is naturally suited for localized manipulation. The framework consists of three main stages:

  1. RoI Extraction: To understand where to edit, the system generates a text description of the scene using a multimodal model (BLIP-2). An LLM (GPT-3.5 Turbo) then compares this description with the user's instruction to identify the specific "Region of Interest" (RoI).
  2. 3D Gaussian RoI Alignment: The system uses 2D grounding tools (Grounding DINO and SAM) to find the RoI in rendered images. It then "lifts" these 2D masks into 3D space by assigning a learnable "RoI attribute" to each 3D Gaussian. This creates a 3D mask that identifies exactly which Gaussians belong to the object being edited.
  3. Delicate Editing: During optimization, the system uses a 2D diffusion model (InstructPix2Pix) to guide appearance changes. Crucially, gradients are only applied to the Gaussians within the RoI. This ensures the rest of the 3D scene remains bit-perfectly identical to the original.
The GaussianEditor pipeline showing RoI-based editing and multi-round results.

Experiments and Results

The authors evaluated GaussianEditor on standard benchmarks including Mip-NeRF360, the Instruct-NeRF2NeRF (IN2N) dataset, and the DTU dataset.

Speed and Efficiency

GaussianEditor completes an edit in under 20 minutes on a single V100 GPU. This is more than twice as fast as Instruct-NeRF2NeRF, which typically takes 45 minutes to 2 hours. This speedup is a direct result of the efficient, explicit Gaussian representation.

Quality and Precision

Qualitative tests demonstrate the model's capability for "multi-round" editing (e.g., making a face look like an Orc, then a Vampire) without geometry degradation or background shifting. Quantitatively, on the Mip-NeRF360 bicycle scene, GaussianEditor outperformed previous methods across all metrics, including CLIP Text-Image Direction Similarity (CTIDS) and Image-Image Similarity (IIS). A user study involving 21 participants showed an 87.07% preference for GaussianEditor over the previous leading method.

Discussion and Limitations

While GaussianEditor offers superior precision, its performance is capped by its sub-modules (GPT-3.5, Grounding DINO, and SAM). If the 2D segmentation is noisy, the 3D RoI will be as well. Additionally, while the model excels at texture and minor shape changes, it struggles with drastic geometric editing—such as adding entirely new structures—if there are no Gaussians present in that space initially. Future work will focus on extending this framework to dynamic 3D environments.

Summary updated

Abstract

Recently, impressive results have been achieved in 3D scene editing with text instructions based on a 2D diffusion model. However, current diffusion models primarily generate images by predicting noise in the latent space, and the editing is usually applied to the whole image, which makes it challenging to perform delicate, especially localized, editing for 3D scenes. Inspired by recent 3D Gaussian splatting, we propose a systematic framework, named Gaus-sianEditor, to edit 3D scenes delicately via 3D Gaussians with text instructions. Benefiting from the explicit property of 3D Gaussians, we design a series of techniques to achieve delicate editing. Specifically, we first extract the region of interest (RoI) corresponding to the text instruction, aligning it to 3D Gaussians. The Gaussian RoI is further used to control the editing process. Our framework can achieve more delicate and precise editing of 3D scenes than previous methods while enjoying much faster training speed, i.e. within 20 minutes on a single V100 GPU, more than twice as fast as Instruct-NeRF2NeRF (45 minutes - 2 hours) 1 1 The editing time varies in different scenes according to the scene structure complexity.. The project page is at GaussianEditor. github.io.
BibTeX
@misc{wang2023gaussianeditor,
  author = {Junjie Wang and Jiemin Fang and Xiaopeng Zhang and Lingxi Xie and Qi Tian},
  title = {{GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions}},
  year = {2023},
  eprint = {2311.16037},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.16037},
}
Figure