View-consistent Object Removal in Radiance Fields

Yiren Lu, Jing Ma, Yu Yin

Abstract

Radiance Fields (RFs) have emerged as a crucial technology for 3D scene representation, enabling the synthesis of novel views with remarkable realism. However, as RFs become more widely used, the need for effective editing techniques that maintain coherence across different perspectives becomes evident. Current methods primarily depend on per-frame 2D image inpainting, which often fails to maintain consistency across views, thus compromising the realism of edited RF scenes. In this work, we introduce a novel RF editing pipeline that significantly enhances consistency by requiring the inpainting of only a single reference image. This image is then projected across multiple views using a depth-based approach, effectively reducing the inconsistencies observed with per-frame inpainting. To accommodate realistic variations in lighting and viewpoint, our pipeline adjusts the appearance of the projected views by generating multiple directional variants of the inpainted image. Additionally, we present an effective and robust multi-view object segmentation approach as a valuable byproduct of our pipeline. Extensive experiments demonstrate that our method significantly surpasses existing frameworks in maintaining content consistency across views and enhancing visual quality.

Introduction

Radiance Fields (RFs), including Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3D-GS), have revolutionized 3D scene representation. Despite their success in novel view synthesis, editing these scenes—specifically removing unwanted objects—remains a significant challenge. Traditional approaches typically rely on 2D image inpainting, where the target object is masked in every training frame and filled using generative models. However, because 2D inpainters are stochastic, they often produce inconsistent results across different frames (e.g., filling a hole with a brick wall in one frame and a wooden fence in another). When these inconsistent images are used to train a 3D model, the result is a "floaty," flickering reconstruction with significant artifacts. This paper addresses the "view-consistency" problem by proposing a pipeline that ensures filled-in content remains stable across all perspectives while respecting the scene's original lighting.

Methodology

The proposed pipeline shifts the focus from multi-view inpainting to a single-view projection strategy, ensuring geometric consistency by design.

1. Reference View Selection and Inpainting

The system automatically identifies a "central" camera pose that maximizes overlap with other views. Within this reference view, the object to be removed is isolated using the Segment Anything Model (SAM) and filled using Stable Diffusion. To ensure the new content integrates geometrically, the pipeline utilizes "Depth-Anything" (a monocular depth model) aligned with sparse COLMAP data to generate a smooth, high-fidelity depth map for the reference frame.

2. Consistent Multi-View Projection

Rather than inpainting every view independently, the system projects the pixels from the single inpainted reference image into all other camera views using the calculated depth map. This approach guarantees that the background content is identical across the entire dataset, eliminating the stochastic variance that plagues per-frame methods.

3. Directional Variants and Occlusion Correction

Standard projection assumes photometric consistency (static lighting), which is rarely true in real-world scenes. To address this, the authors introduce a Directional Variants Module. By querying the original (pre-removal) RF, the system extracts view-dependent lighting information and applies it to the inpainted textures, tailoring the appearance to match the specific lighting of each target view.

Furthermore, an occlusion correction mechanism—utilizing a Z-buffer and depth-prior check—ensures that newly revealed background pixels do not accidentally overwrite foreground objects that should remain in front of the inpainted area.

Experimental Results

The method was evaluated on the SPIn-NeRF dataset, which consists of 10 indoor and outdoor scenes. Each scene includes 60 training images containing the object and 40 ground-truth images where the object was physically removed for evaluation.

Quantitative Performance

The proposed method demonstrated superior performance across both geometric and photometric metrics:

  • Geometric Consistency: Using LoFTR feature matching, the method achieved 319.04 correspondences between views, significantly higher than the previous state-of-the-art, NeRFiller (201.34), and OR-NeRF (34.48).
  • Image Quality: On the SPIn-NeRF benchmark, the NeRF-based implementation achieved a PSNR of 20.82, outperforming all major baselines including LaMask and SPIn-NeRF.

Qualitative Impact

Qualitatively, the pipeline eliminates the "ghosting" and blurriness typical of multi-view inpainting. The resulting textures appear sharp and remain "glued" to the background geometry as the camera moves, providing a much higher level of visual fidelity than previous methods.

Discussion and Limitations

By forcing the 3D model to learn from a single, projected source of truth, this method ensures that geometric consistency is a structural guarantee rather than a learned byproduct. However, the approach is inherently dependent on the quality of the initial 2D inpainting; if the reference image contains hallucinations, those errors will be propagated throughout the scene. Additionally, while the Z-buffer handles most occlusions, extremely complex scenes with thin, overlapping structures (like foliage) may still present challenges for projection. Future work may involve integrating video-based priors to further refine the directional variants and handle more extreme lighting shifts.

Summary updated

Abstract

Radiance Fields (RFs) have emerged as a crucial technology for 3D scene representation, enabling the synthesis of novel views with remarkable realism. However, as RFs become more widely used, the need for effective editing techniques that maintain coherence across different perspectives becomes evident. Current methods primarily depend on per-frame 2D image inpainting, which often fails to maintain consistency across views, thus compromising the realism of edited RF scenes. In this work, we introduce a novel RF editing pipeline that significantly enhances consistency by requiring the inpainting of only a single reference image. This image is then projected across multiple views using a depth-based approach, effectively reducing the inconsistencies observed with per-frame inpainting. However, projections typically assume photometric consistency across views, which is often impractical in real-world settings. To accommodate realistic variations in lighting and viewpoint, our pipeline adjusts the appearance of the projected views by generating multiple directional variants of the inpainted image, thereby adapting to different photometric conditions. Additionally, we present an effective and robust multi-view object segmentation approach as a valuable byproduct of our pipeline. Extensive experiments demonstrate that our method significantly surpasses existing frameworks in maintaining content consistency across views and enhancing visual quality. More results are available at https://vulab-ai.github.io/View-consistent_Object_Removal_in_Radiance_Fields.
BibTeX
@inproceedings{lu2024view,
  author = {Yiren Lu and Jing Ma and Yu Yin},
  title = {{View-consistent Object Removal in Radiance Fields}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681090},
}
Figure