3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting

Qihang Zhang, Yinghao Xu, Chaoyang Wang, Hsin-Ying Lee, Gordon Wetzstein, Bolei Zhou, Ceyuan Yang

Introduction

Traditional image editing has long been confined to the 2D plane. While modern software allows for impressive manipulations, these tools often lack "3D awareness." For instance, moving a subject in a photograph typically leaves a visual "hole" that must be manually filled, and changing the camera perspective is impossible without additional geometric data. Existing 3D editing methods often struggle with disentanglement—either editing the entire scene globally or focusing on isolated objects without environmental context.

3DitScene addresses these challenges by transforming a single 2D image into a fully editable 3D scene. By leveraging Gaussian Splatting and distilling language features into 3D space, the framework enables users to perform complex, 3D-consistent edits using simple text prompts.

3DitScene enables complex 3D-consistent edits from a single 2D image, such as rotating objects or the camera while maintaining background integrity.

Methodology

The 3DitScene framework bridges the gap between 2D imagery and 3D manipulation through four primary technical innovations:

1. Language-Guided 3D Gaussian Splatting (3DGS)

The scene is represented using millions of 3D Gaussians. Each Gaussian is augmented with a language embedding distilled from CLIP. This semantic enrichment allows the model to identify specific clusters of Gaussians as distinct objects, such as "a corgi" or "a headscarf," based on natural language.

2. 3DGS Expansion via Inpainting

To handle novel camera perspectives, the system must account for regions occluded in the original 2D view. 3DitScene utilizes Stable Diffusion’s inpainting capabilities alongside diffusion-based depth estimation (GeoWizard) to realistically "hallucinate" these missing 3D regions.

3. Layout Augmentation

A key challenge in 3D reconstruction from a single image is learning what lies behind visible objects. During the optimization process, 3DitScene employs "Layout Augmentation," where objects are randomly moved or removed. This forces the generative priors to fill in occluded areas, ensuring that the background is correctly rendered even when objects are manipulated later.

4. Unified Optimization

The framework optimizes the scene using a dual-loss approach: a reconstruction loss to maintain fidelity to the original image, and Score Distillation Sampling (SDS) from Stable Diffusion to ensure that newly generated views remain high-quality and realistic.

Results and Evaluation

3DitScene was evaluated against several baselines, including AnyDoor, Object 3DIT, and LucidDreamer, using a diverse set of real-world and AI-generated images.

Quantitative Performance

In a user study, 3DitScene was overwhelmingly preferred for both consistency and image quality. It captured 65.4% of the votes for consistency and 64.0% for image quality, significantly outperforming its competitors.

Qualitative Versatility

The framework demonstrates remarkable success in "chained" edits—sequences of manipulations that were previously difficult to achieve. Examples include moving a subject, deleting it, and then rotating the camera to view the newly revealed background.

Conclusion

3DitScene represents a significant step forward in 3D-aware image editing. By fusing 3D geometry with language semantics and employing clever training signals like layout augmentation, it allows for intuitive, text-guided manipulation of complex scenes. While limitations remain regarding processing time and the handling of complex geometries like reflective surfaces, the method proves that a single high-quality 2D image can serve as a robust foundation for a fully editable 3D proxy.

Summary updated

BibTeX
@inproceedings{zhang2025ditscene,
  author = {Qihang Zhang and Yinghao Xu and Chaoyang Wang and Hsin-Ying Lee and Gordon Wetzstein and Bolei Zhou and Ceyuan Yang},
  title = {{3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=iKDbLpVgQc},
}
Figure