Reversibility-Gated Densification for Dynamic Gaussian Splatting

Gaussian splatting reconstructs a scene from photos as a cloud of small, colored, semi-transparent 3D blobs ("Gaussians"). During training, the method adds new blobs wherever the rendered image disagrees with the photo. This growth step is called densification. It works well for static scenes. In dynamic scenes it fails in a predictable way. When something moves, or when the model's estimate of motion is slightly wrong, the image error is large. The optimizer then spawns blobs to patch that error instead of fixing the motion, and the patches linger as floating debris or jittery artifacts. We propose a test that any candidate blob must pass before it is allowed to exist. Take the candidate's position at a given time, carry it backward through the learned motion to a shared reference pose, then carry it forward again. If it does not land where it started, the motion model cannot explain it. It is then a patch over a transient, a shadow, or a motion error, so it is not created, and the gradient is pushed into the motion model instead.

The appeal is that the test needs no 3D labels. Dense 3D ground truth for moving, deforming scenes is nearly impossible to collect. Methods in this area are trained only by comparing rendered images with real ones, so they have no direct way to tell a real surface from a convenient hallucination. A round-trip residual is a label-free, per-blob measurement of whether the reconstructed 3D structure is consistent with the reconstructed motion. We use it both to control growth and as a diagnostic we can check against the few datasets that do carry 3D ground truth.

Core Claim

In deformable Gaussian splatting, a large share of densification compensates for deformation error rather than adding real structure. Gating densification on forward–backward round-trip consistency of the learned deformation should produce fewer primitives, fewer floaters, and more accurate 3D trajectories, at equal or better rendering quality. The prediction is strongest under fast motion and when real moving objects coexist with unexplainable appearance changes such as moving shadows, flicker, or brief distractors.

The closest work uses reversibility in other ways. Neural Hamiltonian Deformation Fields for Dynamic Scene Rendering (NeHaD) builds reversibility into the motion of an existing Gaussian set through a learned energy function and a symplectic integrator. As the file notes, however, it does not use reversibility to decide which Gaussians to split or clone. RobustSplat attacks floaters by delaying densification and bootstrapping 2D transient masks, but it works on static scenes with distractors and has no motion model. DC4GS observes that standard densification is "blind" because it looks only at gradient magnitude, and gates splitting on directional coherence of gradients within one frame. Our gate instead asks a question across time: does this candidate survive a trip through the motion field?

The Mechanism, and a Design Choice That Matters

A subtle point decides whether the idea works at all. If the deformation is a strict bijection, the round trip is exactly zero by construction and the test carries no information. Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera (NDR) uses such an invertible network, and so does any NDR-style design. NDR's ablation shows invertibility is valuable as a regularizer: error jumped "from 5.29 to 473.25 in one test case" when it was removed. For a diagnostic, though, invertibility removes the signal.

We therefore learn two separate networks. One is a backward field $B(x,t)$ mapping a point observed at time $t$ to the shared reference (canonical) space. The other is a forward field $F(y,t)$ mapping a canonical point to time $t$. A soft cycle loss ties them together. This follows the dual Lagrangian/Eulerian design of MovingParts, which uses reciprocal consistency to prevent degenerate tracking, and the cyclic rollouts of Neural Eulerian Scene Flow Fields (EulerFlow). Because the two fields agree only where the data supports a coherent motion, their disagreement becomes informative.

For a candidate Gaussian $i$ with observed-space centre $x_i(t)$, let $V_i$ be the set of training times at which it is visible, judged by its rendered transmittance. We define its reversibility residual as

$$ r_i = \frac{1}{|V_i|}\sum_{t \in V_i} \frac{\lVert F(B(x_i(t),t),t) - x_i(t) \rVert}{\sigma_i} \;+\; \lambda\,\frac{1}{|V_i|^2}\sum_{t,t' \in V_i} \frac{\lVert B(x_i(t),t) - B(x_i(t'),t') \rVert}{\sigma_i}, $$

where $\sigma_i$ is the Gaussian's scale, so residuals are measured in units of its own size. The second term checks that the same primitive, seen at two times, maps to one canonical location. $\lambda$ weights that term.

The densification rule keeps the usual gradient-magnitude trigger and adds the condition $r_i < \tau$. When a high-gradient candidate fails the gate, it is not split. Its gradient is instead routed to the deformation networks for the next few iterations. Restricting $V_i$ to visible times prevents the gate from punishing objects that leave the field of view or become occluded, the failure mode the file flags for EulerFlow and for reversibility generally. The residual is computed only at densification steps, on candidates only, so its cost should be comparable to DC4GS's reported "roughly 48ms of additional overhead per iteration".

Research Program

We will implement the gate on two deformable splatting backbones. The first is 4D Gaussian Splatting, which deforms canonical Gaussians with a HexPlane encoder and a small decoder. The second is SC-GS, which drives motion through about 512 sparse control points under an as-rigid-as-possible loss. On each backbone we add a backward field and the soft cycle loss.

The work proceeds in three stages. First, a measurement study tests the premise without changing training. We train vanilla backbones, log every densification event, and ask what fraction of spawned Gaussians later end up near-transparent, floating, or with high residual. We also check whether residual predicts 3D trajectory error on scenes that have 3D tracks. If high-residual primitives are not disproportionately bad, the project stops here.

Second, we turn on the gate and study the thresholds, including $\tau$ and whether the "fail" branch reroutes gradient or simply prunes.

Third, we study the residual as an unsupervised quality estimate. Does the per-scene mean residual rank methods the same way 3D ground truth does? This is the contribution aimed at 3D ground-truth scarcity: a usable check on 3D correctness when no 3D labels exist.

Baselines

The naive reference point is the unmodified backbone with standard magnitude-triggered densification. On D-NeRF synthetic data, 4D Gaussian Splatting reports 34.05 dB PSNR. A second naive point disables densification after initialization. RobustSplat found this removes floaters but "leads to overly smooth results". Our gate must beat it on detail metrics, specifically LPIPS on high-frequency regions, not just on floaters.

The strongest dynamic reconstruction competitors on D-NeRF are SC-GS and NeHaD. SC-GS reports 43.31 PSNR. NeHaD reports 40.91 and lists SC-GS at 40.26, so reported numbers for the same baseline disagree by about 3 dB. We will rerun both under one protocol rather than quote either number. "Beating" means a gated SC-GS that matches or exceeds rerun SC-GS PSNR with fewer primitives and lower 3D tracking error. Comparing against NeHaD also isolates whether reversibility helps more as a growth gate than as a built-in motion constraint.

For transient handling, the baseline is RobustSplat (delayed densification plus mask bootstrapping). It reports better PSNR, SSIM and LPIPS than SpotLessSplats, WildGaussians and T-3DGS on NeRF On-the-go and RobustNeRF, but no reported absolute numbers were found on Lacuna for RobustSplat. We will port its delay schedule and masking onto the same deformable backbone.

For gating, the baseline is DC4GS directional-coherence control, which reports "20% to 30%" fewer Gaussians than AbsGS. All methods get the same iteration count, the same initialization, and the same hyperparameter search budget (an equal number of trials per method on a held-out scene).

Three ablations isolate the key ingredient. In the first, the cycle loss is kept but the gate is removed; this tests whether gating matters beyond the regularization effect. In the second, the gate is replaced by a random gate rejecting the same fraction of candidates, which separates the benefit of growing selectively from the benefit of simply having fewer primitives. In the third, an invertible deformation replaces the two-network design. Here the residual should collapse to zero and the gate should do nothing, which confirms the role of the design choice above.

Why not just…?

Why not just delay densification, as RobustSplat does? This is the most serious threat for static scenes with distractors, where waiting for geometry to settle may be all that is needed. Delay works because transients and static structure separate over time in a fixed world frame. In a dynamic scene the "settled" geometry is itself moving, and RobustSplat has no motion model to say which residuals are legitimate. We expect the two methods to tie on NeRF On-the-go and RobustNeRF, and the gate to win on dynamic scenes with injected distractors. If it does not win there, delay is the better tool and our claim fails.

Why not just mask moving content with a 2D video segmenter? Split4D shows that 2D-tracker-based decomposition degrades under fast motion and contact. On the Multi-Human dataset, its tracker-dependent baseline SA4D reaches 0.592 mIoU versus 0.893 for native 4D decomposition. More basically, a segmenter says what moves, not whether a given blob is a correct explanation of that motion, so it cannot catch blobs that patch deformation errors on a legitimately moving object. We will still run a mask-filtered baseline, since it is cheap and may suffice for pure distractor removal.

Why not just gate on temporal support, densifying only where the error comes from many frames? This is a real competitor. It may match our gate whenever transients are brief. It breaks when a real object is fast-moving or briefly visible, since such an object has low support in any one canonical region. It also misses persistent but unexplainable effects such as moving shadows, which have high support. We include a canonical-space support gate as a baseline and stratify results by motion speed and distractor type. If the support gate matches ours across strata, the reversibility machinery is unjustified and we will report that.

Why not just add a global cycle loss and keep standard densification? The "cycle loss, no gate" ablation tests exactly this. NeHaD's improvement over its 4DGS baseline (40.91 vs 35.34 dB as reported there) shows that motion constraints help. It does not show whether unconstrained growth still adds debris on top of a constrained motion field, which is the gap this ablation measures.

Proposed Experiments

Rendering quality uses D-NeRF synthetic (PSNR, SSIM, LPIPS, primitive count, FPS), DyNeRF/Neu3D, and HyperNeRF "vrig". No reported numbers were found on Lacuna for 4D-GS or NeHaD on HyperNeRF and DyNeRF, so we will run them ourselves.

3D correctness without our own labels uses Kubric, which provides 3D tracks. We report average 3D trajectory error of Gaussian centres traced through $F$, and the correlation between per-primitive residual and true 3D error. MovingParts reports "over 85%" mIoU there for part discovery, a secondary check on whether gating preserves decomposable motion.

The stress test builds dynamic-plus-distractor sequences. We composite moving shadows, brief objects that enter and leave the view, and illumination flicker into D-NeRF and Kubric renders. Transient and real content are then known exactly, and floater volume can be measured against the clean geometry.

Static distractor transfer uses NeRF On-the-go and RobustNeRF, comparing against RobustSplat under the same backbone.

Three results decide the project. The measurement study must show that high-residual spawned Gaussians are disproportionately floaters or carry high 3D error. Without that, the premise is false. On the stress test, the gate must beat the temporal-support gate and delayed densification on Kubric 3D error and on floater volume, at matched or better LPIPS. Finally, the random-gate and invertible-deformation ablations must lose. Otherwise the gain comes from having fewer primitives or from the regularizer, not from the reversibility test.