3D Gaussian Splatting (3DGS) represents a scene as millions of small colored, semi-transparent blobs that are optimized until their renderings match a set of photos. The method assumes the world held still while the photos were taken. Casual captures break that assumption. When a pedestrian walks through a few frames, the optimizer spawns opaque blobs to explain them. After the person leaves, those blobs stay behind as floaters hanging in mid-air. Current fixes such as delayed densification, patch-wise transient masks, and robust losses all live inside the training loop. They do nothing for the large and growing stock of splats that have already been trained, shared, or shipped without them.
We propose a post-hoc auditor for a converged, frozen splat. It labels each Gaussian as well-supported static structure, weakly observed but legitimate static structure, or a transient artifact, and it does this without retraining. The idea fits in two sentences. At a converged optimum the photometric gradients from all training views cancel for every Gaussian, but for different reasons. A static blob receives small, agreeing gradients from every view. A transient blob is the scene of a tug-of-war: the few views it was fit to pull hard to keep it, and the views that saw the background behind it pull hard to delete it. Measuring how large these opposing per-view gradients are, as opposed to only their sum, separates transients from rare static objects. Error-based filters confuse the two, because both look inconsistent.
Core Claim
Fix a frozen splat with Gaussians indexed by $i$ and training views indexed by $j$. Let $g_{ij} = \partial \mathcal{L}_j / \partial \alpha_i$ be the gradient of view $j$'s photometric loss with respect to Gaussian $i$'s opacity. A negative value means view $j$ wants the Gaussian more opaque, so we call that view supporting. A positive value means view $j$ wants it gone, so we call that view contradicting. We define a conflict score
$$C_i = \frac{\sum_j |g_{ij}| - \left|\sum_j g_{ij}\right|}{\sum_j |g_{ij}| + \epsilon},$$
which is near 1 when large per-view gradients cancel and near 0 when the views agree. We also define an evidence mass $E_i = \sum_j |g_{ij}|$, normalized by the Gaussian's rendered footprint. This mass plays the role of epistemic support: it measures how much data constrains the Gaussian at all.
The claim has two parts. First, transient floaters have high $C_i$, while rare static objects have low $E_i$ but also low $C_i$, because the views that do not see them stay silent rather than contradicting them. Second, computing the same score on color (spherical-harmonic) gradients instead of opacity gradients isolates a third failure mode. Specular or lighting-varying static surfaces show appearance conflict without existence conflict. NeRF On-the-go and RobustSplat both report that their methods misclassify such surfaces as distractors.
The auditor costs one forward and one backward render per training view, and accumulation into per-Gaussian statistics is streaming. It needs gradient support, so it is an offline audit rather than an on-device monitor. It never changes the weights it measures.
Relation to Prior Work
The method is a direct descendant of PUP 3D-GS. PUP computes a Fisher-information sensitivity score per Gaussian on a pretrained static model and prunes up to 90% of primitives for compression. PUP aggregates sensitivity across views and asks how much a Gaussian matters. We decompose the same per-view gradient signal and ask whether the views agree about the Gaussian. That question PUP never poses, since it assumes a static scene and targets redundancy, not dynamics. A transient floater can carry high Fisher information, because removing it changes the views it was fit to. PUP would therefore keep exactly the Gaussians we want to remove. This is a testable prediction, and it is our main ablation.
The decomposition into evidence and conflict borrows its logic from post-hoc uncertainty work on frozen networks. X-EviProbe separates epistemic evidence (distance to training data) from aleatoric evidence (proximity to known errors) without retraining. CUPID separates the two with a sidecar module. Gradient-based uncertainty for monocular depth shows that a single backward pass of a consistency loss through a frozen network yields useful uncertainty maps. We take that last paper's mechanism, gradients of a consistency objective on frozen weights, and replace the flipped-image consistency with consistency across training views. In a reconstruction this is the natural notion, and it requires no added module.
The in-training uncertainty methods motivate the gap without filling it. NeRF On-the-go trains a DINOv2-driven uncertainty head jointly with the radiance field. UA-4DGS and USplat4D gate regularization by per-Gaussian uncertainty during optimization of dynamic scenes. Stochastic Gaussian Splatting reparameterizes every primitive as a distribution, and its authors note that it does not extend to dynamic scenes. None of these can be pointed at an existing deterministic splat.
DGGS does include an inference-time pruning step. That step relies on binary masks from reference-view inconsistency, refined by 2D segmentation, rather than on a continuous per-primitive score. 3DGS-HPC constrains the design. It documents the rare-object problem, in which static objects seen in one or two views get discarded by error-based filters. Separating low evidence from high conflict is aimed at that failure.
Research Program
The first stage is the auditor itself. We render every training view through the frozen splat and backpropagate each view's L1-plus-SSIM loss separately to opacity and color parameters. We accumulate $\sum_j |g_{ij}|$ and $\sum_j g_{ij}$ per Gaussian in running buffers, so memory stays proportional to the number of Gaussians rather than to Gaussians times views.
Two refinements will be tested rather than assumed. The first replaces signed opacity gradients with a finite-difference counterfactual. For the top-scoring candidates, we zero their opacity in spatial clusters and measure the per-view loss change. This checks whether the first-order signal survives the nonlinearity of alpha blending. The second smooths scores over a spatial k-nearest-neighbor graph. Transients are spatially coherent clumps, and neighboring Gaussians from one densification burst should share a label.
The second stage is repair. We prune Gaussians above a conflict threshold, and we pick the threshold by inspecting the empirical distribution of $C_i$ rather than by tuning on test views. An optional short refinement of a few hundred iterations with densification disabled then closes small seams. We report results both with and without this refinement, so the purely post-hoc claim stays clean. Large holes left by background regions that were occluded in every view are out of scope. The auditor will mark them, through low evidence, rather than invent content.
The third stage is a modest extension to dynamic splats. For a frozen monocular 4D model such as MoSca or Shape-of-Motion on DyCheck, we compute the same conflict score with timesteps in place of views. The question is whether high-conflict dynamic Gaussians coincide with the regions where USplat4D's in-training uncertainty says motion is poorly constrained. This is a diagnostic study, not a headline result, given how hard monocular evaluation is shown to be in Monocular Dynamic View Synthesis: A Reality Check.
Baselines
The naive reference is the frozen vanilla 3DGS model trained on distractor-laden images and left untouched. Every repair must beat it on clean held-out test views. 3DGS-HPC reports that training-time distractor handling improves PSNR by 1.39 to 5.54 dB over native 3DGS on the RobustNeRF data, which bounds the headroom available.
The heuristic post-hoc baseline prunes by low opacity and abnormal scale on the same frozen model, at matched pruning fractions. PUP 3D-GS and RobustSplat both argue this fails because overfit transients are dense and opaque. We expect it to remove detail before it removes floaters.
The principled post-hoc baseline is PUP 3D-GS's Fisher sensitivity score, applied to the same frozen model at matched pruning fractions. We run it in two ways: inverted, removing low-Fisher Gaussians as PUP does, and in our ranking direction. This is the ablation that isolates the key ingredient. If the aggregate Fisher score separates transients as well as the conflict score does, then decomposing per-view gradients adds nothing.
The residual-lifting baseline is the closest simple competitor. It computes per-pixel residuals of the frozen model on training views, thresholds them, and deletes Gaussians whose blending weight concentrates in high-residual pixels. In spirit it is a continuous version of DGGS's mask-based pruning. It uses the same renders and compute as our auditor and differs only in using residual magnitude without gradient sign or cross-view cancellation.
The strongest existing method is a model retrained from scratch with a training-time distractor method on the same images. We use RobustSplat, which its paper reports as best on every NeRF On-the-go scene against SpotLessSplats, WildGaussians, and T-3DGS; the specific per-scene values were not extracted, so no reported result is found on Lacuna for its per-scene PSNR. We also use 3DGS-HPC, at the 13 to 17 minutes on an RTX 4090 that paper reports. For RobustNeRF's real cluttered scenes, the reported reference is 30.87 dB PSNR for RobustNeRF against 25.22 dB for Mip-NeRF 360 (RobustNeRF). On NeRF On-the-go's Patio scene, NeRF On-the-go reports 21.41 dB.
All post-hoc methods share one frozen checkpoint per scene, the same training views, and a threshold rule fixed on two development scenes. Retraining baselines use their published settings, and we report their wall-clock cost. Beating the post-hoc baselines means higher clean-view PSNR and higher primitive-level transient-detection AUROC at every matched pruning fraction. Against retraining, success means recovering at least the majority of the PSNR gap between vanilla 3DGS and the retrained method at a small fraction of retraining time. Full parity is not required for the auditor to be useful on models whose training pipeline is unavailable.
Why not just…?
Why not just retrain with RobustSplat or 3DGS-HPC? When the images and pipeline are available and 15 minutes of GPU time is cheap, retraining is likely to produce the best static reconstruction. This is a real threat to any claim of superior quality, and we do not make one. The auditor's case rests on three settings where retraining is not the answer. The first is splats that exist without their training pipeline. The second is auditing a model already trained with a robust method, to find what it missed. The third is the per-Gaussian diagnosis of rare-object versus transient versus specular, which none of the training-time methods expose. The comparison above quantifies how much of the retraining gain the auditor recovers. If that fraction is small, the paper becomes a diagnostic study rather than a repair method.
Why not just mask dynamic objects with a 2D segmenter before training? 3DGS-HPC reports that semantic segmenters miss unmodeled transients such as shadows, moved chairs, and reflections, and that mask errors in a single frame propagate into the geometry. Segmentation is also a pre-training intervention, so it cannot help an existing splat. On the RobustNeRF scenes, whose distractors are deliberately non-semantic clutter, we expect it to leave visible floaters that the conflict score catches.
Why not just lift high training residuals onto the Gaussians? This is the most serious simple alternative, and it is included as a full baseline rather than dismissed. Our argument against it is mechanistic. In the views a transient was fit to, the residual is low, because the model fit it. In the other views, a high residual pixel may be caused by a wrong background Gaussian rather than by the floater in front of it. Only the gradient sign says whether removing a particular Gaussian would help. If residual lifting matches our AUROC, the cancellation structure is unnecessary, and we will report that.
Why not just prune by opacity, scale, or PUP's Fisher score? RobustSplat's analysis indicates that overfit transients are opaque and well integrated, and PUP shows that opacity heuristics destroy fine detail under aggressive pruning. Fisher sensitivity measures how much a Gaussian matters, and a floater matters a great deal to the views that created it. The matched-fraction ablation directly tests whether conflict adds information beyond total sensitivity.
Proposed Experiments
The main benchmarks are the RobustNeRF dataset, which includes synthetic Kubric scenes and real indoor clutter with clean test views, and the NeRF On-the-go dataset, which covers outdoor pedestrians at varying occlusion levels. We report PSNR, SSIM, and LPIPS on clean test views, plus Gaussian count before and after pruning.
For primitive-level evaluation, we build oracle labels on the Kubric scenes, where distractor geometry is known. A Gaussian cluster counts as transient if zeroing it lowers the error on clean held-out views, and as static otherwise. We then report AUROC and precision at matched pruning fractions for every scoring rule.
The decisive check for the rare-object claim uses static objects visible in at most two training views. On Kubric we insert such objects deliberately, and on real scenes we hand-annotate a small set. The auditor passes if these objects fall in the low-evidence, low-conflict region and survive pruning, while residual lifting and Fisher scoring delete them at comparable recall on true transients.
The decisive check for the specular claim uses glass and metal regions in the On-the-go scenes, which the On-the-go and RobustSplat authors flag as failure cases. The prediction is that these regions show high color conflict but low opacity conflict. If opacity conflict is also high there, the existence-versus-appearance separation fails, and we will say so.
A final sanity check runs the auditor on splats trained on distractor-free captures, which should yield almost no high-conflict Gaussians and negligible PSNR change after pruning. It also runs the auditor on RobustSplat outputs, to measure the residual floaters that training-time methods leave behind. The DyCheck diagnostic is reported separately as exploratory, with the masked-PSNR references of 15.25 for UA-4DGS and 19.63 for USplat4D (at 2x resolution) cited only as context, not as targets.
References
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting
- X-EviProbe: Post-hoc Parameter-Free Evidential Uncertainty Quantification for Frozen Graph Neural Networks
- CUPID: A Plug-in Framework for Joint Aleatoric and Epistemic Uncertainty Estimation with a Single Model
- Gradient-Based Uncertainty for Monocular Depth Estimation
- 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization
- Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction
- Modeling Uncertainty for Gaussian Splatting
- Distractor-free Generalizable 3D Gaussian Splatting
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- Monocular Dynamic View Synthesis: A Reality Check
- RobustNeRF: Ignoring Distractors with Robust Losses