Fast 3D reconstruction networks can turn a handful of photos into a renderable 3D scene in a fraction of a second. They fail badly when someone walks through the shot. These networks find depth by checking which depth makes the pixels in different photos agree. A pedestrian visible in only one photo breaks that agreement, and the network responds by planting floating blobs of color in mid-air.
We propose to treat that disagreement as the signal for finding transient objects instead of as noise. The network already compares every photo against every other photo at every candidate depth, so it can ask, inside the same computation, which photos are outvoted by the others at each pixel. Pixels that the other views consistently contradict are transient. They are down-weighted while the depth is being estimated, and their 3D primitives are the first to be discarded when the scene is compressed to a fixed rendering budget. The result would be a single-pass reconstructor that removes passers-by, shadows and moved objects without a separate segmentation network. Because it outputs a compact scene, it should keep both reconstruction latency and rendering frame rate close to those of an unmodified fast reconstructor.
Core Claim
Some background terms first. A feed-forward reconstructor is a network trained once on many scenes that maps a few images to a 3D scene in one forward pass, with no per-scene optimization. 3D Gaussian Splatting represents a scene as many small colored, semi-transparent ellipsoids that can be rasterized at hundreds of frames per second. A cost volume is a stack recording, for each pixel and each candidate depth, how well the reference image matches the other images when warped to that depth.
MVSplat builds such a cost volume from sparse views and ties each Gaussian's opacity to matching confidence. It reports 26.39 dB PSNR on RealEstate10K at roughly 44 ms per scene. However, it averages evidence across source views. Under distractors its PSNR on the RobustNeRF "Statue" scene drops to 15.45 dB, as reported in Distractor-free Generalizable 3D Gaussian Splatting (DGGS).
Our claim is that replacing this average with a learned consensus aggregation gives most of the robustness of a dedicated distractor-removal pipeline at almost none of its cost. The consensus aggregation estimates, per pixel and per source view, whether that view agrees with the majority across depth planes. Concretely, for reference view $i$, source view $j$, pixel $p$ and depth plane $d$, let $c_{ij}(p,d)$ be the matching correlation. We compute an agreement weight $w_{ij}(p)$ from how closely view $j$'s correlation profile over depth matches the median profile across all other source views. The aggregated cost is
$$C_i(p,d) = \frac{\sum_j w_{ij}(p)\, c_{ij}(p,d)}{\sum_j w_{ij}(p) + \epsilon},$$
where $\epsilon$ is a small constant for numerical stability. A second output is a per-pixel static confidence $s_i(p)$, which captures whether the reference pixel itself has any depth at which the others agree. This confidence multiplies Gaussian opacity and serves as the ranking key when the scene is pruned to a fixed primitive count.
The method is new in two respects. Transient detection lives inside the stereo-matching step rather than in a separate semantic stage. And distractor removal and primitive budgeting become a single mechanism rather than two.
Relation to Prior Work
The closest work is DGGS, the first feed-forward distractor-free Gaussian splatting method. It combines multi-view projection consistency with pretrained segmentation models, scores reference views to pick clean ones, and prunes distractor Gaussians in a second inference stage. It reaches 21.02 dB on Statue. Its authors and our research file both note that the segmentation models and two-stage process add latency over vanilla feed-forward splatting, and the file contains no reported inference latency for it.
We keep DGGS's insight that multi-view inconsistency identifies distractors. We differ in three ways: we compute that inconsistency from the cost volume the network already builds, we use no segmentation model, and we run in one pass. If DGGS's segmentation stage is what carries its accuracy, our proposal fails. That is a falsifiable difference rather than a cosmetic one.
Per-scene optimization methods supply our training intuitions but not our mechanism. NeRF in the Wild learns a per-pixel uncertainty that down-weights transient residuals. RobustNeRF trims the highest-error half of pixels and smooths decisions over 16×16 patches. NeRF On-the-go and WildGaussians predict uncertainty from DINOv2 features. 3DGS-HPC classifies patches by calibrating DINOv2 thresholds with a color-error mixture model. DeSplat assigns distractors to per-view Gaussians, and RobustSplat delays densification until static geometry stabilizes. All of them need minutes to hours per scene because they detect transients from residuals that only exist during optimization. We need a signal available in a single pass, and cross-view disagreement at matching time is one.
These works also constrain us. RobustNeRF and 3DGS-HPC show that per-pixel decisions are noisy, so our weights are predicted at patch resolution. NeRF On-the-go and WildGaussians report that specular surfaces get mistaken for distractors, and a consensus test will inherit that failure.
Budgeted feed-forward splatting supplies the compaction machinery. EcoSplat learns importance rankings that keep 5% of primitives, reaching 24.72 dB on RealEstate10K, about 1000 FPS and 27 MB per scene. It assumes static input. We reuse its budgeted pruning but rank primitives by static confidence times importance, so transients are pruned before redundant static detail.
The dynamic-scene line takes the opposite path. DrivingScene reconstructs moving objects with ego-motion compensation and a residual flow network over two frames. Split4D separates static and dynamic Gaussians during optimization. We deliberately do not reconstruct the moving content. Our goal is a clean static scene from unconstrained captures. We do expose the per-view transient map as an output that a later 4D model could consume.
Research Program
We will build on MVSplat's architecture. Its average over source views is replaced by the consensus aggregation above, which is a small network taking the stack of per-view correlation profiles and producing $w_{ij}$ and $s_i$. The method requires at least three input views, since two views cannot outvote each other. We treat this as a prediction to test rather than hide.
Training uses RealEstate10K scenes with composited distractors, following the synthetic protocol used by DGGS, RobustNeRF and NeRF On-the-go. We paste COCO and Kubric objects into some source views and, critically, add synthetic cast shadows and displaced background objects that no "person" detector would flag. Supervision is a rendering loss against the clean target view. We also add an auxiliary loss on $s$ wherever composite masks are known. An ablation will check whether the auxiliary mask loss is necessary or whether rendering loss alone teaches the consensus.
The pruning stage then keeps the top-$K$ Gaussians by $s \cdot$ importance, with $K$ set as a fraction of pixels as in EcoSplat.
We will analyze where the consensus breaks. The expected cases are distractors present in most views (a parked car), which the file flags via DeSplat as an open failure for every method; specular surfaces; and static objects seen by only one view, the rare-object failure noted for 3DGS-HPC. We will measure each failure mode separately rather than average it away.
Baselines
The naive reference point is MVSplat run unchanged on distractor-laden inputs, using its released weights and also retrained on our augmented data. Its reported 15.45 dB on Statue (from DGGS) is the floor we expect to reproduce. Beating it is necessary but uninteresting.
The strongest existing competitor is DGGS. It is the only feed-forward distractor-free method in the literature we found, it reports 21.02 dB PSNR on Statue, and it uses the same kind of multi-view consistency cue we do, plus segmentation. We will run it with its released configuration on identical view sets and resolutions, on the same GPU, and time it end to end, including segmentation. Its latency has no reported result found on Lacuna, so we will measure it ourselves. Beating DGGS means matching its PSNR within about 0.5 dB at a clearly lower end-to-end latency, or exceeding it on non-semantic distractors such as shadows and moved furniture.
A second strong reference is a segmentation-masked MVSplat. It runs an off-the-shelf segmenter, masks detected movable classes, and feeds the masked images to MVSplat retrained with masked inputs. It gets the same augmented training data and its timing includes the segmenter.
For the latency axis, EcoSplat at matched primitive budgets is the reference for frame rate and size. We report rendering FPS and megabytes alongside PSNR at 5%, 10% and 25% budgets.
Two ablations isolate our key ingredient. First, the same architecture and training data with plain mean aggregation, which isolates consensus weighting from data augmentation. Second, our model with budget pruning ranked by importance alone rather than by static confidence times importance, which isolates the coupling of distractor removal to compaction.
Per-scene methods (WildGaussians, RobustSplat, 3DGS-HPC) are reported as an upper reference on NeRF On-the-go with their training time stated. We do not expect a sparse-view feed-forward model to beat them; the comparison shows the price of skipping optimization.
Why not just…?
Why not just retrain MVSplat on distractor-augmented data and let it learn robustness implicitly? This is the most serious threat. A cost-volume network with a refinement network after it might learn to ignore outlier views without any explicit consensus step. The mean-aggregation ablation, trained on exactly the same augmented data, settles it. If that ablation comes within about 0.5 dB of our model on RobustNeRF and NeRF On-the-go, the contribution is the data recipe and not the mechanism, and we would report it that way. Our reason to expect a gap is structural. Once views are averaged, a single corrupted view shifts the correlation peak for the whole pixel, and later layers cannot tell which view caused the shift.
Why not just mask movable objects with an off-the-shelf segmenter before reconstruction? 3DGS-HPC argues that semantic segmenters miss unmodeled distractors like cast shadows and moved chairs. Split4D notes that flickering 2D masks propagate errors into 3D with no correction. DGGS acknowledges that segmentation adds inference latency. Our shadow and displaced-object subset of the synthetic benchmark tests the first claim directly. The timing comparison tests the latency claim.
Why not just use DGGS? DGGS is the right comparison and may well win on accuracy for people-like distractors, because the segmenter adds semantic knowledge we lack. Our bet is narrower. The consistency signal DGGS computes in a second stage is available, nearly for free, inside the cost volume. Reading it there removes the segmenter's latency and its class bias. If DGGS dominates on both accuracy and the non-semantic subset, the proposal's premise is wrong.
Why not just prune with EcoSplat's importance ranking and hope transients get dropped? EcoSplat states that it assumes static scenes. Its importance signal reflects contribution to rendering, and confidently placed transient floaters can contribute a lot. The importance-only pruning ablation measures whether transients survive compaction without our static-confidence term.
Proposed Experiments
The main evaluation trains on augmented RealEstate10K and tests in four settings. The first is a held-out composite RealEstate10K set with ground-truth masks, where we report PSNR, SSIM and LPIPS on clean targets plus transient-mask IoU. The second is the RobustNeRF dataset, including Statue, against DGGS's 21.02 dB. The third is the NeRF On-the-go scenes, tested zero-shot and stratified by occlusion ratio (5–30%). The fourth is clean RealEstate10K and zero-shot ACID, to confirm no regression against MVSplat's 26.39 dB on clean data. Every row also reports end-to-end reconstruction time, rendering FPS, Gaussian count and storage at the chosen budget.
Three checks decide the outcome. The first is whether consensus aggregation beats mean aggregation on identical augmented data. The second is whether the method matches DGGS on Statue at materially lower latency and beats it on the shadow and moved-object subset. The third is whether accuracy rises with view count as the voting argument predicts: near-baseline at two views, improving at three to six. A flat view-count curve would mean the model is not doing consensus at all. We will also report, separately, the failure rates on stationary distractors, specular surfaces and single-view static objects, so the limits of a consistency-based detector are measured rather than asserted.
References
- MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-view Images
- Distractor-free Generalizable 3D Gaussian Splatting
- NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
- RobustNeRF: Ignoring Distractors with Robust Losses
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- WildGaussians: 3D Gaussian Splatting In the Wild
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- DeSplat: Decomposed Gaussian Splatting for Distractor-Free Rendering
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- EcoSplat: Efficiency-controllable Feed-forward 3D Gaussian Splatting from Multi-view Images
- DrivingScene: A Multi-Task Online Feed-Forward 3D Gaussian Splatting Method for Dynamic Driving Scenes
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation