Introduction
Traditional image editing has long been confined to the 2D plane. While modern software allows for impressive manipulations, these tools often lack "3D awareness." For instance, moving a subject in a photograph typically leaves a visual "hole" that must be manually filled, and changing the camera perspective is impossible without additional geometric data. Existing 3D editing methods often struggle with disentanglement—either editing the entire scene globally or focusing on isolated objects without environmental context.
3DitScene addresses these challenges by transforming a single 2D image into a fully editable 3D scene. By leveraging Gaussian Splatting and distilling language features into 3D space, the framework enables users to perform complex, 3D-consistent edits using simple text prompts.
Methodology
The 3DitScene framework bridges the gap between 2D imagery and 3D manipulation through four primary technical innovations:
1. Language-Guided 3D Gaussian Splatting (3DGS)
The scene is represented using millions of 3D Gaussians. Each Gaussian is augmented with a language embedding distilled from CLIP. This semantic enrichment allows the model to identify specific clusters of Gaussians as distinct objects, such as "a corgi" or "a headscarf," based on natural language.
2. 3DGS Expansion via Inpainting
To handle novel camera perspectives, the system must account for regions occluded in the original 2D view. 3DitScene utilizes Stable Diffusion’s inpainting capabilities alongside diffusion-based depth estimation (GeoWizard) to realistically "hallucinate" these missing 3D regions.
3. Layout Augmentation
A key challenge in 3D reconstruction from a single image is learning what lies behind visible objects. During the optimization process, 3DitScene employs "Layout Augmentation," where objects are randomly moved or removed. This forces the generative priors to fill in occluded areas, ensuring that the background is correctly rendered even when objects are manipulated later.
4. Unified Optimization
The framework optimizes the scene using a dual-loss approach: a reconstruction loss to maintain fidelity to the original image, and Score Distillation Sampling (SDS) from Stable Diffusion to ensure that newly generated views remain high-quality and realistic.
Results and Evaluation
3DitScene was evaluated against several baselines, including AnyDoor, Object 3DIT, and LucidDreamer, using a diverse set of real-world and AI-generated images.
Quantitative Performance
In a user study, 3DitScene was overwhelmingly preferred for both consistency and image quality. It captured 65.4% of the votes for consistency and 64.0% for image quality, significantly outperforming its competitors.
Qualitative Versatility
The framework demonstrates remarkable success in "chained" edits—sequences of manipulations that were previously difficult to achieve. Examples include moving a subject, deleting it, and then rotating the camera to view the newly revealed background.
Conclusion
3DitScene represents a significant step forward in 3D-aware image editing. By fusing 3D geometry with language semantics and employing clever training signals like layout augmentation, it allows for intuitive, text-guided manipulation of complex scenes. While limitations remain regarding processing time and the handling of complex geometries like reflective surfaces, the method proves that a single high-quality 2D image can serve as a robust foundation for a fully editable 3D proxy.
Summary updated