Abstract
3D Gaussian Inpainting with Depth-Guided Cross-View Consistency (3DGIC) is a novel framework designed to remove objects from 3D scenes while maintaining strict multi-view consistency. Traditional 2D inpainting applied per-frame often results in "floaters" and flickering due to geometric misalignment. 3DGIC addresses this by first refining inpainting masks using rendered depth to project visible background information from alternative views, thereby minimizing the region requiring generative synthesis. Subsequently, it uses a 2D-inpainted reference view to anchor the optimization of new 3D Gaussians. Experimental results on the SPIn-NeRF dataset demonstrate that 3DGIC achieves state-of-the-art FID scores and superior visual fidelity compared to existing NeRF and 3DGS-based methods.
Introduction
Removing objects from 3D scenes is a fundamental task in computer vision and graphics, yet it remains challenging due to the requirement of multi-view consistency. When an object is removed, the "hallucinated" background must match perfectly across every camera angle. Inconsistent inpainting leads to blurry artifacts, "floaters" (unconnected geometry in 3D space), and distracting flickering during camera movement. While NeRF-based methods have attempted to solve this, they are often slow and struggle with sharp details. 3D Gaussian Splatting (3DGS) offers speed and high fidelity, but naive inpainting within the 3DGS framework often leaves holes or inconsistent textures.
Methodology
The 3DGIC framework operates in two distinct stages to ensure that the filled-in background is both realistic and geometrically sound.
1. Inferring Depth-Guided Inpainting Masks
Instead of treating the entire 2D object mask as a region to be entirely hallucinated, 3DGIC leverages the existing 3D scene geometry. By using rendered depth maps, the system projects pixels from other viewpoints into the current view's masked region. This process identifies portions of the background that are occluded in one view but visible in another. By filling these areas with "real" pixels rather than generated ones, the system significantly reduces the size of the mask that requires generative inpainting, preserving original background details.
2. Inpainting-Guided 3DGS Refinement
To fill the remaining truly occluded regions, the system selects a "reference view"—typically the one with the largest remaining mask. A 2D inpainter (such as LAMA or Stable Diffusion LDM) is used to generate RGB and depth data for this view. These 2D results are then projected into 3D space as a set of colored point clouds, which act as a geometric anchor. New 3D Gaussians are optimized against this anchor, ensuring that the newly synthesized content is consistent across all rendered views.
Experiments and Results
The authors evaluated 3DGIC on several challenging real-world datasets, including the SPIn-NeRF benchmark and various 360° scenes from InNeRF360 and Mip-NeRF360. Initial object masks were generated automatically using the Segment Anything Model (SAM).
Quantitative Performance
On the SPIn-NeRF dataset, 3DGIC (utilizing LDM as the 2D inpainter) achieved a state-of-the-art FID score of 36.4, significantly outperforming previous methods such as GScream (38.6) and MALD-NeRF (44.9). Lower FID scores indicate that the generated images are more realistic and closer to the distribution of real-world scenes.
Qualitative Analysis
Visual comparisons demonstrate that 3DGIC successfully avoids the "black holes" and heavy shadows common in SPIn-NeRF, as well as the blurry, inconsistent textures produced by Gaussian Grouping. The method excels at preserving fine background details, such as electrical sockets or complex table textures, which are often smudged or erased by competing approaches.
Conclusion and Future Work
3DGIC provides a robust solution for 3D scene inpainting by combining the speed of 3D Gaussian Splatting with a depth-guided consistency mechanism. By prioritizing real background pixels over hallucinated ones and using a reference view anchor, it achieves superior visual and geometric consistency.
However, the method's performance is dependent on the accuracy of the initial 3DGS-rendered depth maps. Future work may focus on improving depth estimation from sparse views and enhancing the robustness of the initial segmentation masks for smaller objects.
Summary updated