Photo collections of public places almost always contain people, cars and barriers that move between shots. 3D reconstruction methods, such as radiance fields (neural networks that map 3D points to color and density) and Gaussian splatting (scenes built from many small colored 3D blobs), handle these "transient occluders" by detecting them and ignoring their pixels. That works only when some other photo sees the hidden surface. When a stretch of wall is covered in every view, or in nearly every view of a sparse capture, the reconstruction is left with a hole. The current fix asks a large pretrained image generator to paint in the missing background. That generator has never seen this building, so it paints a plausible generic wall and the 3D model then copies it as if it were real.
This note proposes a different source for the missing pixels: a small generative model trained only on the parts of the scene that several views agree are static. Facades, pavements and fountains repeat their own textures heavily, so the rest of the scene is usually the best available evidence for what lies behind a tourist. We train a local patch-level denoising model on these verified background patches, with occluder pixels excluded from training, and use it to fill holes inside the 3D optimization. Only the hole's 3D primitives are updated, and the surrounding reconstruction holds them in place. The claim is testable and limited. For holes whose texture recurs elsewhere in the scene, a prior built from that scene's verified inliers should reconstruct the hidden surface more faithfully than a foundation inpainter. For holes whose texture appears nowhere else, no prior can help, and we expect to show that too.
Core Claim
Two findings from separate literatures fit together. In generative inverse problems, whole-image priors push the global structure of their training data onto the measurements. Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems shows that a 64×64 patch diffusion model, refined self-supervised on the target data, beats a whole-image model trained on the correct dataset (30.34 vs 29.87 dB deblurring PSNR). In 2D inversion, IntereStyle shows that forcing a generator to explain an off-distribution obstacle, such as a microphone, distorts the nearby in-distribution content, so a chin turns into a beard. That finding carries over to prior construction: a scene-specific prior trained on images that still contain occluders will learn the occluders. RI3D takes exactly this risk. It personalizes repair and inpainting diffusion models leave-one-out on the raw input views, and it was evaluated only on clean scenes.
Our core claim is that the useful ingredient is where the prior's training data comes from: patches that multi-view robust fitting has already certified as static. Model capacity is secondary. Distractor-removal methods already compute a per-pixel inlier signal, and this signal tells us both what to train the prior on and where it is allowed to act. We treat the choice of architecture as an experimental variable. One option is a small patch model trained from scratch; the other is a low-rank fine-tune of a foundation inpainter on the same inlier crops. We would accept the result if the second option won.
Research Program
The pipeline starts from a robust Gaussian-splatting reconstruction. We use the patch-wise residual classification of 3DGS-HPC, which trains in roughly 13–17 minutes, and also try the trimmed loss of RobustNeRF as an alternative front end. From this run we keep each pixel's inlier weight. Each Gaussian gets a support score: the inlier-weighted count of views in which it is visible. Gaussians with low support, together with empty regions bounded by well-supported geometry, define the "holes."
Next we train the scene prior. We extract 64×64 patches from training images only where every pixel has a high inlier weight, and we add flips and small photometric jitter. A patch diffusion model is trained on them, following the patch-prior recipe of Hu et al. The model never sees a pixel classified as transient. This choice is the main difference from RI3D's personalization and from SparseGS-W, which segments tourists with EVF-SAM and fills them using a fine-tuned Stable Diffusion inpainter whose output then serves as hard pseudo-ground truth.
Filling happens in a render, denoise, update loop. We render novel and training views that look into each hole, add noise to the hole pixels at an intermediate level, and denoise them with the patch prior. The surrounding well-supported pixels are held fixed as measurements, in the style of measurement-guided sampling. The denoised patches serve as soft targets, but only for hole Gaussians; well-supported Gaussians receive no gradient from the prior. Noise levels are annealed over rounds. Consistency across views comes from the shared 3D representation, which averages the targets from all views. We do not try to make independent 2D fills agree with each other. Following BIGPrior, which learns a pixel-wise gate between observed data and prior hallucination, we also output a 3D map recording how much each rendered pixel was set by the prior versus by observations. This map is a reported deliverable and is not used for training.
Finally, we analyze when the method should fail. Feedback-guided Data Synthesis for Imbalanced Classification argues that guidance cannot produce features missing from a model's training data, and that argument applies to our scene prior as well. We therefore score each hole for texture recurrence, defined as the distance in a fixed feature space between the true hidden patches and their nearest neighbors among the scene's inlier patches. All results are reported against this score.
Baselines
The do-nothing reference is the robust reconstruction alone, with holes left as they come out (3DGS-HPC or RobustNeRF). Any fill must beat it on the hole metrics. A fill could also lose to it on perceptual-fidelity grounds, if it invents structure.
The strongest existing method for this exact problem is SparseGS-W's occlusion-handling pipeline. It produces segmentation masks, fills them with a fine-tuned Stable Diffusion inpainter, and then trains on the inpainted images as targets. The paper reports 19.01 dB PSNR and 48 FID on PhotoTourism with 5 views, against about 14.8 dB and 159 for the previous best 3DGS method. It is the strongest competitor because it is the only cited system that fills transient holes with a generative prior in-the-wild. For a fair comparison we give it the same poses, the same initialization and the same hole masks (both its own and ours), and we report the 5-view protocol where it was defined. RI3D's two-stage repair and inpaint, with leave-one-out personalization on raw views, is the second generative competitor. It tests whether personalization without inlier filtering is enough. We run it with matched GPU-hours.
For dense-capture distractor removal, the reported references are RobustNeRF at 30.87 dB (versus 25.22 for Mip-NeRF 360) on its benchmark, and NeRF On-the-go at 21.41 dB on Patio. Neither paper reports PSNR restricted to permanently occluded regions; no reported result found on Lacuna for hole-region fidelity. That is why we build the hole benchmark below.
Three ablations isolate the key ingredient. The first trains the same patch model on all training pixels, occluders included, to test the IntereStyle-style contamination claim directly. The second replaces the scene prior with a generic patch prior trained on a broad natural-image set. The third applies the scene prior as hard pseudo-ground truth on all Gaussians rather than soft guidance on hole Gaussians only. Every method gets the same per-scene GPU budget, and we report the cost of training the prior separately. Beating SparseGS-W means lower masked LPIPS and higher masked PSNR on high-recurrence holes, with whole-image metrics no worse.
Why not just…?
Why not just mask with a segmenter and inpaint with Stable Diffusion, as SparseGS-W does? This is the main hypothesis under test. Hu et al. and BIGPrior show that whole-image priors favor plausibility over fidelity when the content is mismatched. A one-off historical facade is mismatched content for a general inpainter, and once SparseGS-W uses the inpainted image as a target, the error is fixed in 3D. The masked-region comparison stratified by texture recurrence settles the question. If SparseGS-W matches us on high-recurrence holes, our claim fails.
Why not just LoRA-fine-tune the foundation inpainter on the scene's clean regions? This is a real threat, and it may work as well as or better than a small patch model. We therefore run it as a co-equal variant on identical inlier crops. If it wins, the contribution narrows to what we consider the essential finding, namely that inlier-filtered scene data is what helps, and the method uses the larger model. The run that fine-tunes on occluder-contaminated data separates the two explanations.
Why not just rely on robust losses and accept the holes? RobustNeRF acknowledges that trimming discards data, and 3DGS-HPC, WildGaussians and DGGS all report holes and speckle where coverage is missing. The do-nothing baseline measures this directly. Our method must improve the holes without lowering metrics on supported regions, which should be guaranteed if supported Gaussians receive no prior gradient.
Why not use an untrained Deep Image Prior on the renders? DIP carries no distribution bias. The FMPlug authors (Saving Foundation Flow-Matching Priors for Inverse Problems) note, however, that it lacks structural knowledge and converges slowly. We include DIP-on-render fills at matched compute. We expect DIP to be competitive on low-recurrence holes, where scene data cannot help, and weaker on repeated textures.
Proposed Experiments
The decisive benchmark is a controlled permanent-occlusion set built from the paired captures of the RobustNeRF benchmark and the clean test views of NeRF On-the-go. In each scene we drop or edit training views so that a chosen surface region is covered in all remaining views, either by real distractors or by composited people. We then score fills against held-out clean views using PSNR, SSIM and LPIPS restricted to the hole mask. Synthetic Kubric scenes add a fully controlled condition: we texture the hidden surface either with a texture repeated elsewhere in the scene or with a unique, deliberately unusual texture. This gives the high- and low-recurrence extremes directly.
On PhotoTourism we run the SparseGS-W 5-view protocol, report whole-image PSNR and FID against its published 19.01 dB and 48, and add masked metrics on held-out views where the tourist-covered region happens to be clear.
The decisive pattern has three parts. Our method must beat SparseGS-W and generic-prior fills on high-recurrence holes. The occluder-contaminated prior must do measurably worse, showing that inlier filtering matters. On low-recurrence holes every prior, ours included, should converge toward DIP-level quality. If the gain does not shrink as recurrence falls, then the scene-evidence explanation is wrong and some other effect is responsible.
A secondary check targets the reflective-surface false positives noted by NeRF On-the-go and WildGaussians. If specular regions fall below the support threshold and are refilled with matte texture, our method will show it as a fidelity drop on glass and water. We will report this as a limitation rather than tune it away.
References
- Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems
- IntereStyle: Encoding an Interest Region for Robust StyleGAN Inversion
- RI3D: Few-Shot Gaussian Splatting With Repair and Inpainting Diffusion Priors
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- RobustNeRF: Ignoring Distractors with Robust Losses
- SparseGS-W: Sparse-View Gaussian Splatting for Unconstrained Image Collections with Diffusion Priors
- BIGPrior: Toward Decoupling Learned Prior Hallucination and Data Fidelity in Image Restoration
- Feedback-guided Data Synthesis for Imbalanced Classification
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- Saving Foundation Flow-Matching Priors for Inverse Problems