Scene representations built from 3D Gaussians, which are small colored blobs rendered by sorting and splatting them onto the image, now render dynamic video at hundreds to thousands of frames per second. They get there by deleting primitives. The rules for which primitives to delete, however, come from three separate literatures that do not agree with each other. One rule deletes blobs that look inconsistent across cameras. Another deletes blobs whose opacity flickers over time. A third deletes blobs that barely affect the rendered colors. None of these rules can tell a "floater" (a spurious blob hanging in mid-air that satisfies one camera and contradicts others) from a real object that happens to be seen by only one or two cameras. None of them decides which surviving blobs are actually static and could skip the per-frame motion computation, which is the next latency bottleneck once rasterization becomes fast.
We propose one cheap per-Gaussian statistic that makes all three decisions. The optimizer already computes it: how much the gradients coming from different cameras agree on whether a Gaussian should become more or less opaque. A real surface receives gradients that agree, or receives none from cameras where it is hidden. A floater is pulled up by one camera and pushed down by the others. We compute this agreement once across cameras at the same instant and once across time. That separates Gaussians into three groups: floaters to prune, dynamic content to keep deforming, and static content that can be frozen out of the deformation network. The claim is that this single test can replace three incompatible pruning rules and a 2D segmentation model, protect rarely seen static objects, and lower end-to-end frame time, not just rasterization time.
Core Claim
Let $g_{i,v,t}$ be the gradient of the photometric loss with respect to the opacity of Gaussian $i$, computed from camera $v$ at timestep $t$. This gradient is already weighted by how visible the Gaussian is to that camera, so hidden Gaussians contribute nothing. We define within-instant agreement and across-time agreement as
$$A^{\text{view}}_i = \frac{\sum_t \left|\sum_v g_{i,v,t}\right|}{\sum_t \sum_v |g_{i,v,t}| + \epsilon}, \qquad A^{\text{time}}_i = \frac{\left|\sum_{t}\sum_v g_{i,v,t}\right|}{\sum_t \left|\sum_v g_{i,v,t}\right| + \epsilon},$$
where $\epsilon$ is a small constant. Both scores lie in $[0,1]$. $A^{\text{view}}$ asks whether simultaneous cameras agree about Gaussian $i$. $A^{\text{time}}$ asks whether different moments agree about it once each moment's cameras have been pooled. Both are exponential moving averages over training, so they cost one accumulation per backward pass.
Each combination of scores has a reading. Low $A^{\text{view}}$ marks a floater, which we prune. High $A^{\text{view}}$ with low $A^{\text{time}}$ marks genuine dynamic content, which keeps its deformation. High $A^{\text{view}}$ with high $A^{\text{time}}$ marks static content, which is moved to a time-invariant branch that never queries the deformation field.
A rare static object seen by one camera gets near-zero gradients from cameras where it is hidden, not opposing ones, so it keeps high agreement. This is the property that error-based filters lack. Its one weak point is rare objects that are inside other cameras' view and unoccluded but badly lit or out of focus there, and we test that case directly.
Relation to Prior Work
Several papers motivate the proposal by exposing the gap. 1000+ FPS 4D Gaussian Splatting prunes Gaussians whose temporal opacity has a large second derivative and shares key-frame visibility masks across frames. It exceeds 1000 FPS, but a persistent floater has zero temporal derivative and survives this test, a fast-accelerating real object can be wrongly pruned, and the authors report that preprocessing outside rasterization becomes the new bottleneck. SpotlessSplats prunes by gradient magnitude ("utilization"). Our score uses gradient direction across views instead, and magnitude cannot see direction: a floater can be highly "utilized" by the one view it overfits. 3DGS-HPC documents the rare-object failure, in which static objects seen in one or two views are classified as distractors because of their high early error. Our test targets exactly that failure.
Other papers supply ingredients we reuse or argue against. MCGS projects Gaussian centers into several views and prunes those with low DINOv2 feature similarity. That is the closest cross-view pruning idea, but it applies only to static sparse-view scenes, needs a foundation model's features at every view, and scores appearance rather than what the loss actually demands. HGS shows the payoff of routing static content away from temporal parameters: 300+ FPS at 1080p and 6.87 MB models. It gets its static/dynamic split from the Track Anything segmentation model, whose mask errors permanently corrupt geometry, and we replace that model with the agreement test. STDR learns spatio-temporal activity masks during a warm-up phase to reduce ghosting, and DeGauss and DeSplat factorize scenes into static and dynamic (or per-view distractor) Gaussians through learned masks or regularized opacities. All three learn the decomposition as extra parameters. We read it off gradients that already exist.
RobustSplat constrains the design. It shows that densification, the step that splits and clones Gaussians where error is high, turns transient error into permanent floaters, and that disabling densification entirely over-smooths the result. We therefore gate densification on agreement rather than turning it off. The base 4D representation comes from 4D Gaussian Splatting, which pairs canonical Gaussians with a deformation field built from six 2D feature planes (HexPlane).
Research Program
We build the test on top of 4D-GS and keep the backbone unchanged, so that any gain can be attributed to the test.
Step 1: accumulation. Per-view opacity gradients are summed in the rasterizer's backward pass into three running buffers per Gaussian. This adds memory proportional to the number of Gaussians and no extra network evaluations.
Step 2: using the scores. At each densification interval, Gaussians with low $A^{\text{view}}$ are pruned and are not allowed to clone. Gaussians that are static by both scores migrate to a static set whose deformation query is skipped at render time. The migration is soft during a warm-up period, meaning it can be reversed if agreement drops.
Step 3: analysis of whether the scores measure what we claim. On synthetic scenes with ground-truth floaters, dynamic objects, and inserted rare static objects, we check whether the scores separate the three classes, measured as per-class precision and recall of the thresholded scores. We also measure how much this separation depends on camera count, because $A^{\text{view}}$ needs at least two simultaneous views that constrain the same point.
Step 4: the casual single-camera case. For captures where every frame is a different instant, $A^{\text{view}}$ collapses into $A^{\text{time}}$. We then test whether a transient distractor, such as a pedestrian, shows up as low agreement just as a floater does. This is plausible, because the distractor is supported by one frame and contradicted by the frames that see the background behind it. It would extend the test to the RobustNeRF and On-the-go distractor benchmarks.
Baselines
The naive reference point is vanilla 4D-GS with default pruning, run on the same data at the same iteration budget. Its reported figures are 34.05 dB PSNR and 82 FPS at 800×800 on D-NeRF with about 18 MB of storage (4D Gaussian Splatting). Our method must match its quality while cutting end-to-end frame time.
The strongest latency competitor is 4DGS-1K. It reports over 1000 FPS rasterization on N3DV, about 9× faster than vanilla 4DGS, with storage falling from about 2 GB to about 50 MB. No reported PSNR for 4DGS-1K on N3DV was found on Lacuna. We run its public pruning schedule inside the same backbone, at a matched final primitive count and matched pruning ratio. We report end-to-end time including deformation queries and mask computation, not rasterization time alone. Beating it means equal or lower end-to-end frame time together with higher PSNR at equal primitive count, or fewer surviving floaters.
The strongest decomposition competitor is HGS. It reports about 32.36 dB PSNR on N3DV with 300+ FPS at 1080p, and we run it with its Track Anything masks. The stronger quality reference on N3DV is Instant Gaussian Stream, which reports 34.15 dB at 204 FPS (Instant Gaussian Stream). We cite it as a reference but do not match it directly, since it is a streaming method. For distractor removal in the single-camera case, the reference points are 3DGS-HPC, which reports a 1.39 to 5.54 dB gain over 3DGS on RobustNeRF, and SpotlessSplats, which reports 22.81 dB on the On-the-go "Fountain" scene against 21.70 dB for 3DGS. All methods get the same training iteration count and the same GPU (an RTX 3090 for N3DV timing). Each gets an equal hyperparameter sweep of five threshold settings chosen on held-out views.
Four ablations isolate the key ingredient. The first replaces agreement with gradient magnitude, which is utilization pruning. The second replaces it with a per-Gaussian deformation-magnitude threshold. The third replaces it with MCGS-style DINOv2 cosine similarity. The fourth uses only $A^{\text{time}}$, which tests whether simultaneous cameras matter.
Why not just…?
Why not just freeze Gaussians whose learned deformation offsets are near zero? This is the most serious threat to the static-routing half of the proposal, and it may work as well there. It cannot find floaters, though: a floater has zero offset and would be frozen and kept. The deformation-magnitude ablation settles the question. If it matches us on static routing and latency, we narrow our claim to floater and rare-object handling.
Why not just combine 4DGS-1K's temporal pruning with MCGS-style feature consistency? The two rules conflict on accelerating objects, which the temporal rule removes and the feature rule keeps. The feature rule also requires DINOv2 features at every view, which is exactly the preprocessing cost that 4DGS-1K identifies as the new bottleneck. The DINOv2 ablation measures both the quality difference and the added seconds per frame.
Why not just mask moving objects with a 2D tracker, as HGS does? 3DGS-HPC shows that such models label object categories rather than physical transience, and so they miss shadows and moved furniture. HGS's own limitations note that mask leakage permanently distorts geometry. We report HGS on scenes containing shadow and specular motion to show this failure where it occurs, rather than assuming it.
Decisive Checks
We run three experiments, each with a condition under which the idea should be abandoned or narrowed.
The first is a controlled synthetic study built on the D-NeRF scenes and on Sync-NeRF-style multi-camera renders. We inject known floaters and insert static objects visible in exactly one or two cameras. The test must separate floaters from rare objects with higher precision and recall than both gradient magnitude and the temporal second-derivative rule. If rare objects in other cameras' unoccluded view are still pruned, that failure case is real, and we report it as a limit.
The second runs on N3DV and the Google Immersive dataset. We report PSNR, SSIM, and LPIPS; primitive count and storage; the masked total variation flicker metric from Compensating Spatiotemporally Inconsistent Observations; and a per-stage timing breakdown covering deformation query, preprocessing, and rasterization. The prediction is that the fraction of Gaussians routed to the static set tracks the static area of the scene, and that deformation-query time falls in proportion to it.
The third applies the single-camera variant to RobustNeRF and On-the-go. If collapsed agreement does not beat 3DGS by a margin comparable to the reported 3DGS-HPC gains, we restrict the claim to synchronized multi-camera capture. That setting is exactly where the within-instant test has its information.
References
- 1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering
- SpotLessSplats: Ignoring Distractors in 3D Gaussian Splatting
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- MCGS: Multiview Consistency Enhancement for Sparse-View 3D Gaussian Radiance Fields
- HGS: Hybrid Gaussian Splatting with Static-Dynamic Decomposition for Compact Dynamic View Synthesis
- STDR: Spatio-Temporal Decoupling for Real-Time Dynamic Scene Rendering
- DeGauss: Dynamic-Static Decomposition with Gaussian Splatting for Distractor-free 3D Reconstruction
- DeSplat: Decomposed Gaussian Splatting for Distractor-Free Rendering
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
- Instant Gaussian Stream: Fast and Generalizable Streaming of Dynamic Scene Reconstruction via Gaussian Splatting
- Compensating Spatiotemporally Inconsistent Observations for Online Dynamic 3D Gaussian Splatting