Distractors Are the Pixels That Hurt the Rest of the Scene: Cross-View Gradient Conflict for Transient-Free Gaussian Splatting

3D reconstruction methods that turn a set of photos into a renderable scene assume the world holds still. When a pedestrian walks through a few frames, the optimizer tries to explain the pedestrian anyway. In 3D Gaussian Splatting (3DGS), which represents a scene as millions of small colored, semi-transparent blobs, this goes wrong in a specific way. The large errors caused by the pedestrian look to the optimizer like unresolved fine detail, so it spawns new blobs to fit them. Those blobs stay behind as "floaters" after the person has left. Current fixes either threshold the per-pixel color error or ask a large pretrained vision model which pixels look like people and cars. Both approaches share a failure. A real static object seen from only one or two cameras also has high error and weak support from other views, so it gets deleted along with the pedestrian. Shadows, moved furniture, and distractors that share the background's color are missed.

We propose a different test for whether a pixel is a distractor. Ask what would happen to every other training view if the model changed its parameters to fit this pixel. This can be computed from gradients the renderer already produces. It separates three cases that color error alone cannot tell apart. Fitting a transient pixel makes other views worse, so the two gradients oppose. Fitting a pixel on an under-reconstructed static edge makes other views better, so they align. Fitting a pixel on a rarely seen static object leaves other views unchanged, so they are orthogonal. We use this signed score to down-weight distractor pixels in the loss and, more importantly, to keep them out of the statistic that decides where new blobs are spawned.

Core Claim

For pixel $p$ in training view $v$, let $g_{p,i}$ be the gradient of that pixel's photometric loss with respect to the parameters $\theta_i$ of Gaussian $i$ (position, opacity, base color). Let $G_{-v,i}$ be a running average of the gradient that the other training views send to the same Gaussian. Let $D_i$ be the optimizer's per-parameter step scale, which is the square root of Adam's second-moment estimate. The cross-view conflict score is

$$s_p \;=\; \sum_i w_{p,i}\,\big\langle D_i^{-1} g_{p,i},\; G_{-v,i} \big\rangle ,$$

where $w_{p,i}$ is Gaussian $i$'s blending weight along pixel $p$'s ray. Because the actual update is an Adam step $\Delta\theta_i \approx -\eta D_i^{-1} g_{p,i}$, a first-order Taylor expansion gives the change in loss on all other views as $\Delta L_{-v} \approx -\eta\, s_p$. A strongly negative $s_p$ therefore means that fitting the pixel would damage the rest of the scene.

The quantity is signed and relational, and both properties matter. A rarely observed static object sits on Gaussians that other views barely touch, so $G_{-v,i}\approx 0$ and $s_p\approx 0$ regardless of how large its residual is. The score does not flag it. A transient occupies space that other views see as background or empty air. Those views constrain the same Gaussians in the opposite direction, so $s_p$ is large and negative. The score also behaves correctly near convergence. At a stationary point, each view's gradient is balanced by the others. A well-fit static pixel then contributes a small $g_{p,i}$, while a pixel in a real tug-of-war contributes a large inner product.

Where This Sits in Prior Work

The gradient-based uncertainty literature supplies the core intuition, which is to measure how much a model's internal parameters would have to change to accommodate an input. Classification Uncertainty of Deep Neural Networks Based on Gradient Information and Epistemic Uncertainty Quantification for Pretrained Neural Networks score inputs by the magnitude of their loss gradient with respect to network weights. Gradient-Based Uncertainty for Monocular Depth Estimation and Revisiting Gradient-based Uncertainty for Monocular Depth Estimation show that backpropagating a consistency loss to internal features yields pixel-wise uncertainty maps. Their maps track true error better than raw output differences, with an Absolute Relative AUSE of 0.024 versus 0.044 on KITTI. These works motivate our design, but they use gradient magnitude. The research file notes that magnitude cancels or misleads when signals conflict. In our setting magnitude is exactly the wrong statistic, because the rare static object and the transient both produce large gradients. We keep their core idea of measuring re-learning stress and replace the magnitude with a signed alignment against the rest of the data.

Within 3DGS, three papers analyze parameter gradients, but only on static scenes. PUP 3D-GS uses a diagonal Fisher approximation of the loss curvature to prune 90% of Gaussians from trained static models. It is offline and never asks whether the data disagree. Gradient-Direction-Aware Density Control (GDAGS) shows that densification should look at gradient direction as well as magnitude. It defines a Gradient Coherence Ratio over the sub-gradients arriving at each Gaussian and uses it to choose between splitting and cloning. GDAGS provides the closest technical precedent. It pools all contributing pixels at one Gaussian, and it was tested only on static benchmarks. We make the comparison per pixel and across views, which lets the method decide which observation is at fault rather than only noting that a Gaussian is contested. SpotlessSplats also computes color-to-position gradients, but only to prune low-utilization blobs after the fact. It detects distractors with Stable Diffusion features.

The distractor-removal literature both constrains and motivates the design. RobustSplat identifies the densification mechanism behind floaters. It responds with a fixed delay on densification and masks bootstrapped from residuals, and it reports that disabling densification entirely over-smooths the scene. Any fix therefore has to be selective rather than global. RobustNeRF trims the highest residuals. NeRF On-the-go and WildGaussians predict uncertainty from DINOv2 features. 3DGS-HPC fuses a color-residual mixture model with DINOv2 patch cues. It explicitly documents the rare-object failure and the fragility of foundation features on blurry early renders. Bayesian and view-dependent uncertainty for 3DGS, in Modeling Uncertainty for Gaussian Splatting and View-Dependent Uncertainty Estimation of 3D Gaussian Splatting, measures how well a region is covered by views, not whether the observations of it are consistent over time. The first explicitly leaves dynamic scenes to future work.

Research Program

The first component is the score itself, implemented inside the rasterizer's backward pass. Standard 3DGS already forms each pixel's gradient contribution to each Gaussian before summing them. We add one dot product per pixel–Gaussian pair against a stored per-Gaussian vector $G_{-v,i}$. That vector is an exponential moving average of accumulated gradients from recent iterations on other views. It costs one extra parameter-sized buffer and produces a dense per-pixel map without any external network.

There is a chicken-and-egg problem: the score is needed to weight the loss whose gradient defines it. We resolve it with a score-only backward pass that writes no parameter gradients, followed by the weighted training backward. We will measure the wall-clock overhead against the one-pass alternative, which reuses the score from the image's previous visit at low resolution.

Scores are rank-normalized per image and spatially smoothed over small patches, in the spirit of RobustNeRF's smoothing kernel. The result is converted into a soft inlier weight.

The second component is densification gating, which we expect to matter more than the loss mask. The viewspace positional-gradient statistic that triggers splitting and cloning is accumulated only from pixels with non-negative $s_p$. A Gaussian pulled hard by a transient then never reaches the split threshold, while static edges that other views also want refined still do. This replaces RobustSplat's fixed delay with a per-pixel criterion.

The third component is a diagnostic study that asks whether the three predicted regimes actually appear. We will plot $s_p$ against residual magnitude for known transients, known rare static objects, and under-fit static edges, and check whether they separate along the sign of $s_p$. We will also ablate which parameter groups enter the inner product (appearance only, position only, all), whether the Adam preconditioner $D_i^{-1}$ helps compared with plain Euclidean alignment or a PUP-style Fisher metric, and the averaging window for $G_{-v,i}$. The window matters because a stale average compares gradients taken at different parameter values.

The main anticipated failure is global appearance change such as lighting shifts in Photo Tourism, where every pixel of an image conflicts with the others. There we will add per-image appearance embeddings as WildGaussians does and treat the result as a stress test rather than a headline. Real view-dependent effects seen from only two directions can also produce opposing gradients. We will report that case separately rather than hide it.

Baselines

The naive reference point is vanilla 3DGS trained on the cluttered images with standard densification. It shows how much damage distractors do. The second reference is a trimmed-loss 3DGS that discards the top half of per-pixel residuals, as RobustNeRF does for NeRF. It represents "just ignore the worst pixels." On the RobustNeRF benchmark, the RobustNeRF paper reports 30.87 dB PSNR on its distractor scene against 25.22 dB for Mip-NeRF 360. On the NeRF On-the-go Patio scene it reaches 20.54 dB against NeRF On-the-go's 21.41 dB, as reported in NeRF On-the-go.

The key-ingredient ablation holds everything fixed except the score. We run the identical pipeline, including patch smoothing, rank normalization, loss weighting and densification gating, but replace $s_p$ with the per-pixel residual magnitude. We run a second variant that replaces it with the unsigned gradient magnitude $\sum_i w_{p,i}\lVert g_{p,i}\rVert$, which is the direct transplant of the classification-uncertainty literature. These ablations isolate whether the sign and the cross-view reference carry the benefit. They are the comparisons most likely to embarrass the idea, so they get the full tuning budget.

A third learned-free baseline transplants GDAGS's Gradient Coherence Ratio to the distractor setting. Gaussians with low coherence are barred from densifying, and the pixels rendered mostly by such Gaussians are down-weighted. This tests whether per-Gaussian direction statistics are already enough.

A fourth baseline is classical: reproject each pixel through the rendered depth into neighboring views and flag it if the colors disagree after occlusion checking.

The strongest existing methods are RobustSplat, SpotlessSplats (SLS-mlp), 3DGS-HPC and WildGaussians. RobustSplat is the most direct competitor because it targets the same densification mechanism without a semantic model. No reported RobustSplat number was found on Lacuna. The file states only that it achieves the best results across scenes against SpotlessSplats, WildGaussians and T-3DGS. For SpotlessSplats, the file reports 22.81 dB on On-the-go Fountain against 21.70 dB for vanilla 3DGS. For 3DGS-HPC, it reports gains of 1.39 to 5.54 dB over native 3DGS on RobustNeRF, with training of 13–17 minutes on an RTX 4090.

All methods run from the same COLMAP poses, image resolution, iteration count and GPU. Each gets the same small grid search over its main thresholds on one held-out scene per dataset. We report both our reruns and the published figures. Beating RobustSplat means higher mean PSNR and lower LPIPS across all On-the-go and RobustNeRF scenes under this protocol, with a clearly larger margin on the rare-object test described below. Matching it on the standard benchmarks while clearly winning on rare objects would still be a meaningful result.

Why not just…?

Why not just threshold photometric residuals or use a trimmed loss? Residual magnitude cannot separate a rarely seen static object from a transient; 3DGS-HPC names exactly this failure. SpotlessSplats and RobustSplat also report that densification noise and color camouflage corrupt residual-based masks. Still, this is the most serious threat. On scenes dominated by large, high-contrast pedestrians, residuals may be nearly as good as our score. The residual-magnitude ablation, run with identical gating, settles the question. If it matches our score within noise on both the standard benchmarks and the rare-object test, the cross-view term is not earning its cost and the idea fails.

Why not just apply GDAGS's coherence ratio to cluttered scenes? Low coherence at a Gaussian says the Gaussian is contested, not which observation is wrong. A static edge that is being over-fit and a background blob contested by a passing car both show low coherence, and GDAGS's own design treats low coherence as a signal to split, which is the opposite of what a transient needs. The transplanted-coherence baseline tests this directly. We expect it to suppress some floaters but also to block legitimate splitting on fine static detail, which would show up as lower PSNR on textured regions.

Why not just use DINOv2 or Stable Diffusion features? These models find semantic classes such as people and cars. 3DGS-HPC reports that they miss shadows and moved inanimate objects and that they fail on blurry early renders. They also add a large network to every training run. On semantically obvious distractors they may beat us, and we will report that honestly. The rare-object and shadow subsets are where our method should win. A hybrid that multiplies our score with a semantic prior is a natural follow-up, but it is not the claim here.

Why not just delay densification, or check multi-view color consistency by reprojection? RobustSplat's delay is a global schedule, and its authors show that removing densification altogether over-smooths the scene. Our gating decides per pixel and never switches densification off. Reprojection checks depend on rendered depth, and depth is exactly what floaters corrupt early in training. The gradient score works through the renderer's own blending weights and therefore handles occlusion implicitly. Both are included as baselines, and the Gaussian-count and floater comparisons will show whether the extra machinery matters.

Proposed Experiments

The primary real-world evaluations are the twelve NeRF On-the-go sequences and the four RobustNeRF scenes, scored with PSNR, SSIM and LPIPS on clean held-out views. We also report Gaussian count, training time and peak memory, because our method adds a backward pass and must stay near 3DGS-HPC's roughly 2 GB and 13–17 minute budget to be practical. Photo Tourism with appearance embeddings serves as a secondary stress test.

The decisive experiment is a controlled compositing benchmark built on Mip-NeRF 360 captures. Into each capture we insert two kinds of synthetic 3D objects rendered with the true camera poses. The first kind moves between frames, so it is a transient with an exact mask. The second kind is static but placed so that only two or three training cameras see it, which makes it a rare object with an exact mask. We also vary how closely the transient's color matches the background.

On this benchmark we measure distractor detection as per-pixel AUROC and AUPR against the known transient masks, following the uncertainty-estimation literature. We measure rare-object retention as PSNR inside the rare object's footprint in held-out views near its few observing cameras, and floater volume as the number of Gaussians left in regions only transients ever occupied.

The core prediction has two parts. The sign of $s_p$ should separate transients from rare static objects, where residual magnitude, gradient magnitude and per-Gaussian coherence all fail to. The advantage should grow as transients become more color-camouflaged and as rare objects get fewer views. If the signed score does not beat residual magnitude on rare-object retention at matched transient recall, the central claim is false.

A final mechanistic check tracks, for each Gaussian spawned by densification during training, whether its spawning gradient came from pixels later identified as transient. If gating works as intended, the fraction of transient-driven splits should fall sharply relative to vanilla 3DGS and RobustSplat's schedule, while the number of splits on static textured regions stays unchanged.