Visibility-Uncertainty-guided 3D Gaussian Inpainting via Scene Conceptional Learning

mingxuan cui, Qing Guo, Yuyi Wang, Hongkai Yu, Di Lin, Qin Zou, Ming-Ming Cheng, Zequn Qin, Xi LiSubmission to ICLR 2026 · Withdrawn

TL;DR

VISTA is a 3D Gaussian Splatting (3DGS) inpainting framework that seamlessly removes unwanted static objects and dynamic distractors (like swimming fish or pedestrians). By coupling visibility uncertainty across viewpoints with textual-inversion-based scene concept learning, VISTA iteratively combines geometric multi-view visual cues with generative semantic priors to synthesize consistent, artifact-free 3D scenes.

What Problem Does This Paper Address?

While 3D Gaussian Splatting provides real-time, photorealistic novel view synthesis, editing 3D representations—specifically 3D Gaussian Inpainting (3DGI)—remains challenging. When removing an object, two problems arise:

  1. Multi-view visual inconsistency: Standard 2D inpainting applied independently per view ignores the fact that occluded surfaces in one perspective might be fully visible in another.
  2. Semantic context gaps: Large masked areas often lack visual information across all input views, requiring generative hallucination that respects the broader scene style without hallucinating artifacts.

Existing solutions either rely on naive 2D inpainting (which breaks 3D multi-view consistency), depth-map warping (which fails on complex textures and dynamic motions), or ignore moving distractors altogether.

Methods

VISTA addresses these challenges through an iterative loop comprising two complementary modules:

Multi-View Images + Masks
         │
         ▼
   [ VISTA-GI ] ──(Generates Visibility Uncertainty Map U)
         │                   │
         │ (Re-optimizes 3DGS)│ (Guides Conceptual Inversion)
         ▼                   ▼
    3D Gaussian Splats   [ VISTA-CL ] (Textual Inversion + Diffusion)
         ▲                   │
         └──── Iterative ────┘ (Produces refined 2D views)
  • VISTA-GI (Visibility-Uncertainty-Guided 3DGI): For any 3D point $X$, the method projects it into $V$ neighboring camera views and calculates color variance $u_X = \text{var}(\{x_v\})$. High variance marks points that are inconsistent across views (such as moving objects or occlusions). Projecting this uncertainty back into camera frames creates pixel-wise visibility uncertainty maps $U$. During 3DGS optimization, loss functions down-weight pixels with high uncertainty or mask coverage.
  • VISTA-CL (Visibility-Uncertainty-Guided Scene Conceptual Learning): When a region lacks cross-view visibility, generative filling is required. Using textual inversion on a pre-trained latent diffusion model (Stable Diffusion v1.5), VISTA learns a scene concept token $s^*$ conditioned specifically on the unmasked, low-uncertainty scene content. A modified RePaint diffusion process then inpaints the masked areas conditioned on $s^*$.
  • Iterative Framework: Over typically three iterations, refined 2D images $\tilde{\mathcal{I}}$ from VISTA-CL and the original "anchor" images $\mathcal{I}$ are jointly fed into VISTA-GI, progressively eliminating artifacts and refining geometry.
  • Dynamic Distractor Handling: VISTA pairs coarse open-vocabulary tracking masks (via MASA) with its uncertainty maps. Coarse masks catch slow/stationary objects, while uncertainty maps catch fast-moving artifacts.

Datasets

The authors evaluated VISTA on both static and dynamic benchmarks:

  • SPIn-NeRF Dataset: 10 static 3D scenes (3 indoor, 7 outdoor) with 100 images per scene, used to evaluate static object removal with ground truth background reference.
  • Underwater 3D Inpainting Dataset (New): 10 dynamic underwater video scenes derived and resampled from the UTB180 tracking benchmark. Fast-swimming fish serve as moving distractors.
  • NeRF-on-the-Go (Tree scene): Real-world evaluation for dynamic pedestrian removal.
  • Synthetic Extreme-View Dataset: Custom scene evaluating performance under wide angular baselines ($45^\circ$ separation, 8 views).

Key Insights and Ideas

  • Uncertainty as a bridge between geometry and semantics: Color variance across projected neighbor views acts as an unsupervised detector for transient elements and occlusions, telling both the Gaussian optimizer and the diffusion model which pixels to trust.
  • Scene concept personalization beats generic prompt inpainting: Instead of generic text prompts (e.g., "fill with gravel"), optimizing a pseudo-word token $s^*$ via textual inversion captures the exact lighting, texture, and semantics of the background context.
  • Anchoring with raw images: Replacing raw views entirely with diffusion-inpainted views causes drift due to accumulated generative errors. Retaining raw images alongside inpainted views acts as an anchor to preserve scene structure.

Key Results

  • Static Scene Inpainting (SPIn-NeRF): VISTA achieved state-of-the-art performance, outperforming InFusion, Gaussian Grouping, and GScream with 16.48 dB PSNR, 0.59 SSIM, 0.418 LPIPS, and 113.58 FID.
  • Dynamic Underwater Distractor Removal: VISTA improved image quality and semantic coherence (0.51 UCIQE, 1.64 URanker, 0.72 CLIP score) and dominated a 48-image user study with a 71.09% Rank-1 preference (compared to 9.63% for SpotLessSplats and 3.39% for InFusion).
  • Extreme View baselines: Under sparse viewpoints, VISTA maintained 26.38 dB PSNR compared to SpotLessSplats' 24.75 dB and InFusion's 19.34 dB.
  • Computational Efficiency: Reconstructing and inpainting an entire scene took ~33 minutes on a single RTX 4090—comparable to standard 3DGS pipelines and ~10× faster than NeRF-based approaches like SPIn-NeRF (~7.5 hours).

Limitations and Future Work

  • Static Camouflage Failures: If a dynamic object remains nearly stationary across views (low optical flow/uncertainty) and is simultaneously missed by the 2D segmenter, VISTA treats it as a static background structure (e.g., a rock) and fails to remove it.
  • Extremely Sparse Views: When angular separation between cameras is excessively large ($>45^\circ$), neighboring camera baselines cannot compute reliable multi-view consistency, shifting the burden almost entirely onto 2D diffusion inpainting.

Summary updated

Abstract

3D Gaussian Splatting (3DGS) has emerged as a powerful and efficient 3D representation for novel view synthesis. This paper extends 3DGS capabilities to inpainting, where masked objects in a scene are replaced with new contents that blend seamlessly with the surroundings. Unlike 2D image inpainting, 3D Gaussian inpainting (3DGI) faces the challenge of effectively leveraging complementary visual and semantic cues from multiple input views, as occluded areas in one view may be visible in others. To address this, we propose a method that measures the visibility uncertainties of 3D points across different input views and uses them to guide 3DGI in utilizing complementary visual cues. We also employ the uncertainties to learn a semantic concept of the scene without the masked object and use a diffusion model to fill masked objects in the input images based on the learned concept. Finally, we build a novel 3DGI framework VISTA by integrating VISibility-uncerTainty-guided 3DGI with scene conceptuAl learning. VISTA generates high-quality 3DGS models capable of synthesizing artifact-free and naturally inpainted novel views. Furthermore, our approach extends to handling dynamic distractors arising from temporal object changes, enhancing its versatility in diverse scene reconstruction scenarios. We demonstrate the superior performance of our method over state-of-the-art techniques using two challenging datasets: the SPIn-NeRF dataset, featuring 10 diverse static 3D inpainting scenes, and an underwater 3D inpainting dataset derived from UTB180, which includes fast-moving fish as inpainting targets.
BibTeX
@misc{cui2025visibility,
  author = {mingxuan cui and Qing Guo and Yuyi Wang and Hongkai Yu and Di Lin and Qin Zou and Ming-Ming Cheng and Zequn Qin and Xi Li},
  title = {{Visibility-Uncertainty-guided 3D Gaussian Inpainting via Scene Conceptional Learning}},
  year = {2025},
  eprint = {2504.17815},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.17815},
}
Figure