TL;DR
VISTA is a 3D Gaussian Splatting (3DGS) inpainting framework that seamlessly removes unwanted static objects and dynamic distractors (like swimming fish or pedestrians). By coupling visibility uncertainty across viewpoints with textual-inversion-based scene concept learning, VISTA iteratively combines geometric multi-view visual cues with generative semantic priors to synthesize consistent, artifact-free 3D scenes.
What Problem Does This Paper Address?
While 3D Gaussian Splatting provides real-time, photorealistic novel view synthesis, editing 3D representations—specifically 3D Gaussian Inpainting (3DGI)—remains challenging. When removing an object, two problems arise:
- Multi-view visual inconsistency: Standard 2D inpainting applied independently per view ignores the fact that occluded surfaces in one perspective might be fully visible in another.
- Semantic context gaps: Large masked areas often lack visual information across all input views, requiring generative hallucination that respects the broader scene style without hallucinating artifacts.
Existing solutions either rely on naive 2D inpainting (which breaks 3D multi-view consistency), depth-map warping (which fails on complex textures and dynamic motions), or ignore moving distractors altogether.
Methods
VISTA addresses these challenges through an iterative loop comprising two complementary modules:
Multi-View Images + Masks
│
▼
[ VISTA-GI ] ──(Generates Visibility Uncertainty Map U)
│ │
│ (Re-optimizes 3DGS)│ (Guides Conceptual Inversion)
▼ ▼
3D Gaussian Splats [ VISTA-CL ] (Textual Inversion + Diffusion)
▲ │
└──── Iterative ────┘ (Produces refined 2D views)
- VISTA-GI (Visibility-Uncertainty-Guided 3DGI): For any 3D point $X$, the method projects it into $V$ neighboring camera views and calculates color variance $u_X = \text{var}(\{x_v\})$. High variance marks points that are inconsistent across views (such as moving objects or occlusions). Projecting this uncertainty back into camera frames creates pixel-wise visibility uncertainty maps $U$. During 3DGS optimization, loss functions down-weight pixels with high uncertainty or mask coverage.
- VISTA-CL (Visibility-Uncertainty-Guided Scene Conceptual Learning): When a region lacks cross-view visibility, generative filling is required. Using textual inversion on a pre-trained latent diffusion model (Stable Diffusion v1.5), VISTA learns a scene concept token $s^*$ conditioned specifically on the unmasked, low-uncertainty scene content. A modified RePaint diffusion process then inpaints the masked areas conditioned on $s^*$.
- Iterative Framework: Over typically three iterations, refined 2D images $\tilde{\mathcal{I}}$ from VISTA-CL and the original "anchor" images $\mathcal{I}$ are jointly fed into VISTA-GI, progressively eliminating artifacts and refining geometry.
- Dynamic Distractor Handling: VISTA pairs coarse open-vocabulary tracking masks (via MASA) with its uncertainty maps. Coarse masks catch slow/stationary objects, while uncertainty maps catch fast-moving artifacts.
Datasets
The authors evaluated VISTA on both static and dynamic benchmarks:
- SPIn-NeRF Dataset: 10 static 3D scenes (3 indoor, 7 outdoor) with 100 images per scene, used to evaluate static object removal with ground truth background reference.
- Underwater 3D Inpainting Dataset (New): 10 dynamic underwater video scenes derived and resampled from the UTB180 tracking benchmark. Fast-swimming fish serve as moving distractors.
- NeRF-on-the-Go (Tree scene): Real-world evaluation for dynamic pedestrian removal.
- Synthetic Extreme-View Dataset: Custom scene evaluating performance under wide angular baselines ($45^\circ$ separation, 8 views).
Key Insights and Ideas
- Uncertainty as a bridge between geometry and semantics: Color variance across projected neighbor views acts as an unsupervised detector for transient elements and occlusions, telling both the Gaussian optimizer and the diffusion model which pixels to trust.
- Scene concept personalization beats generic prompt inpainting: Instead of generic text prompts (e.g., "fill with gravel"), optimizing a pseudo-word token $s^*$ via textual inversion captures the exact lighting, texture, and semantics of the background context.
- Anchoring with raw images: Replacing raw views entirely with diffusion-inpainted views causes drift due to accumulated generative errors. Retaining raw images alongside inpainted views acts as an anchor to preserve scene structure.
Key Results
- Static Scene Inpainting (SPIn-NeRF): VISTA achieved state-of-the-art performance, outperforming InFusion, Gaussian Grouping, and GScream with 16.48 dB PSNR, 0.59 SSIM, 0.418 LPIPS, and 113.58 FID.
- Dynamic Underwater Distractor Removal: VISTA improved image quality and semantic coherence (0.51 UCIQE, 1.64 URanker, 0.72 CLIP score) and dominated a 48-image user study with a 71.09% Rank-1 preference (compared to 9.63% for SpotLessSplats and 3.39% for InFusion).
- Extreme View baselines: Under sparse viewpoints, VISTA maintained 26.38 dB PSNR compared to SpotLessSplats' 24.75 dB and InFusion's 19.34 dB.
- Computational Efficiency: Reconstructing and inpainting an entire scene took ~33 minutes on a single RTX 4090—comparable to standard 3DGS pipelines and ~10× faster than NeRF-based approaches like SPIn-NeRF (~7.5 hours).
Limitations and Future Work
- Static Camouflage Failures: If a dynamic object remains nearly stationary across views (low optical flow/uncertainty) and is simultaneously missed by the 2D segmenter, VISTA treats it as a static background structure (e.g., a rock) and fails to remove it.
- Extremely Sparse Views: When angular separation between cameras is excessively large ($>45^\circ$), neighboring camera baselines cannot compute reliable multi-view consistency, shifting the burden almost entirely onto 2D diffusion inpainting.
Summary updated