TL;DR
This paper introduces a novel pipeline for removing objects from 3D Radiance Fields (like NeRF and 3D Gaussian Splatting) by inpainting only a single reference image and projecting it across all other views. By combining depth-based projection with a "directional variants" module to handle changing light, the authors achieve significantly better 3D consistency and visual quality than traditional methods that inpaint every frame individually.
What Problem Does This Paper Address?
Radiance Fields (RFs), such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3D-GS), have revolutionized how we represent 3D scenes. However, editing these scenes—specifically removing an unwanted object—is notoriously difficult.
Current approaches usually rely on 2D image inpainting. They mask the object in every training image and ask an AI to "fill in the blanks." The problem? 2D inpainters are stochastic; they might put a brick wall in one frame and a wooden fence in the next. When you combine these inconsistent images to train a 3D model, the result is a "floaty," flickering mess of artifacts. This paper tackles the "view-consistency" problem: how do we ensure that the filled-in content looks the same from every angle and respects the scene's lighting?
Methods
The authors propose a pipeline that shifts the heavy lifting from multi-view inpainting to a single-view projection strategy:
- Reference View Selection: The system automatically picks a "central" camera pose that has the most overlap with other views.
- High-Quality Single-View Processing:
- Masking: Uses Segment Anything Model (SAM) to isolate the object.
- Inpainting: Uses Stable Diffusion to fill the hole in just that one reference image.
- Depth Estimation: Uses "Depth-Anything" (a monocular depth model) aligned with sparse COLMAP data to create a smooth, accurate depth map.
- Consistent Multi-View Projection: Instead of inpainting other views, the system "projects" the pixels from the inpainted reference image into the other camera views using the depth map.
- Directional Variants Module: Standard projection assumes lighting never changes. To handle real-world shadows and reflections, the authors query the original (pre-removal) RF to generate "directional variants" of the reference image, tailoring the appearance of the inpainted area to match the lighting of each target view.
- Occlusion Correction: They implement a Z-buffer and a depth-prior check to ensure that when an object is removed, the "newly revealed" background doesn't accidentally overwrite foreground objects that should be in front of it.
Datasets
The researchers evaluated their method using the SPIn-NeRF dataset, a standard benchmark for 3D inpainting.
- Composition: 10 scenes (indoor and outdoor).
- Data: Each scene includes 60 training images with the unwanted object and 40 ground-truth images where the object was physically removed during capture for evaluation.
- Comparison: They compared their results against major baselines: LaMask, SPIn-NeRF, OR-NeRF, and NeRFiller.
Key Insights and Ideas
- Less is More: The core "aha" moment is that inpainting less leads to better 3D results. By forcing the 3D model to learn from a single, projected source of truth, geometric consistency is guaranteed by design rather than left to chance.
- RFs as Lighting Guides: The authors realized that even if an RF contains an object you want to delete, it still "understands" the lighting of the scene. They use the dirty RF to extract view-dependent information, which they then apply to their clean, inpainted textures.
- Depth Priors over RF Depth: RFs often produce "noisy" depth maps near object boundaries. By using a dedicated monocular depth estimation model (Depth-Anything) and aligning it to sparse 3D points, they get much cleaner projections than if they relied on the NeRF's internal geometry.
Key Results
The method was tested on both NeRF and 3D Gaussian Splatting backends, showing impressive gains:
- Geometric Consistency: Using feature matching (LoFTR), the authors' method found 319.04 correspondences between views, compared to only 201.34 for the previous state-of-the-art (NeRFiller) and a measly 34.48 for OR-NeRF.
- Image Quality: On the SPIn-NeRF dataset, "Ours-NeRF" achieved a PSNR of 20.82, outperforming all baselines.
- Visual Fidelity: Qualitatively, the method eliminates the "ghosting" and "blurriness" common in other methods, producing sharp textures that stay "glued" to the background as the camera moves.
Limitations and Future Work
- Inpainter Dependency: The quality of the final 3D scene is heavily capped by the quality of the initial 2D inpainting. If Stable Diffusion creates a hallucination that doesn't fit the scene's logic, that error is projected everywhere.
- Complex Occlusions: While the Z-buffer helps, extremely complex scenes with many overlapping thin structures (like tree branches) might still struggle with projection gaps.
- Future Directions: The authors suggest that integrating more advanced generative models or video-based priors could further refine the "directional variants" to handle even more extreme lighting shifts.
Summary updated