Rigid Motion as a Lie Detector for 2D Tracks in Real-Time Dynamic Splatting

Systems that rebuild a moving scene from video, and then render it from new viewpoints, usually lean on 2D cues produced by other networks: point tracks that follow a pixel across frames, depth estimates, and segmentation masks. When those cues are wrong, the 3D reconstruction inherits the error and usually cannot undo it. This happens, for example, when a track jumps to a neighboring surface during an occlusion. We propose to use the rigid-body assumption differently from how it is normally used. Usually it serves as a motion prior. Here it would also act as a consistency test on the 2D evidence. A rigid piece of an object has only six degrees of freedom (three for position, three for orientation), yet dozens of tracks land on it. That redundancy tells us which tracks are lying.

The same structure pays for speed. If most of a scene moves as rigid pieces, rendering a frame needs only a lookup of one rigid transform per piece, with no neural network evaluated per primitive. Expensive per-primitive deformation is reserved for the regions where the data proves the motion is genuinely non-rigid. The two-sentence version is this: tracks that disagree with a piece's rigid motion are either tracker errors or signs of real deformation, and re-rendering tells the two apart. Resolving that ambiguity correctly should make dynamic reconstruction robust to bad 2D cues and fast at render time.

Core Claim

Our setting is 3D Gaussian Splatting, which represents a scene as millions of small colored, semi-transparent ellipsoids ("Gaussians") rasterized onto the screen. Its dynamic extensions face a three-way tension.

Unconstrained deformation fields are flexible but slow and unstable. 4D Gaussian Splatting decodes per-Gaussian motion from a factorized space-time feature grid. Deformable 3D Gaussians uses a time-conditioned network. Both show ghosting under fast motion.

Pure rigid transforms are fast but brittle. SplatSim drives segmented Gaussian clusters with physics-engine poses and cannot handle soft objects.

Methods that initialize motion from 2D trackers are efficient, but they are hostage to those trackers. From Tokens to Nodes explicitly reports that drifting tracks or depth irreversibly corrupt its control nodes.

Our claim is that a single decision rule, applied per track and per frame, addresses all three. When a track disagrees with the rigid motion of the cluster it belongs to, we re-render the cluster under its rigid motion and check the photometric error at that track's pixels. If the image is explained, the track was wrong and is down-weighted. If the image is not explained, the cluster is genuinely deforming, and it is either split or granted a residual deformation budget. Densification is allowed only after that, and only for errors that persist after motion has been freed.

What is new is not a rigidity prior, which SC-GS already uses through an as-rigid-as-possible loss on sparse control points. Nor is it a rigid/articulated/soft hierarchy, which HoliGS already builds. What is new is using rigid redundancy to adjudicate the reliability of upstream 2D cues, and letting that adjudication determine where non-rigid capacity, and therefore render-time cost, is spent.

Research Program

The representation consists of canonical Gaussians assigned softly to $K$ clusters. Each cluster $k$ carries a per-frame rigid transform $T_k(t)$. A cluster flagged as non-rigid additionally carries a compact residual, a small set of temporal basis displacements per Gaussian stored as tensors rather than computed by a network. At render time, each Gaussian's position is therefore a table lookup plus at most a short linear combination. That is the property that should let us approach the static-splatting regime instead of the ~20 FPS that HoliGS reports for its RealNVP flow layer.

Track adjudication works as follows. Each 2D track $j$ observed at pixel $u_j(t)$ has a lifted canonical point $X_j$ in cluster $k$. Its rigidity residual is $r_j(t) = \lVert \pi(T_k(t) X_j) - u_j(t) \rVert$, where $\pi$ is camera projection. Its photometric residual $e_j(t)$ is the rendering error in a small patch around $u_j(t)$ when the cluster is rendered rigidly. The cluster motion is fit by weighted least squares on tracks plus a photometric term:

$$ \min_{T_k(t)} \; \sum_j w_j(t)\, r_j(t)^2 \;+\; \lambda \, \mathcal{L}_{\text{photo}}(T_k(t)), \qquad w_j(t) = \exp\!\Big(-\tfrac{r_j(t)^2}{\sigma_r^2}\Big) + \Big(1 - \exp\!\Big(-\tfrac{r_j(t)^2}{\sigma_r^2}\Big)\Big)\,\mathbb{1}\big[e_j(t) > \tau\big]. $$

Here $\lambda$ balances the two terms, $\sigma_r$ sets the scale of an acceptable rigidity residual, and $\tau$ is a photometric threshold.

The weight expresses the 2×2 logic. A track that agrees with the rigid motion keeps its weight. A track that disagrees, at a location the rigid rendering already explains, loses its weight, because the tracker is wrong. A track that disagrees, at a location the rendering fails to explain, keeps its weight but is routed to a split-or-deform step instead of pulling the rigid fit.

That step works as follows. If enough high-weight disagreeing tracks share a coherent alternative rigid motion, found by a RANSAC-style hypothesis over six-degree-of-freedom fits, the cluster splits. This recovers articulation without a skeleton detector, avoiding the 2D pose dependency that HoliGS inherits from PoseNet. Otherwise, the cluster receives residual deformation capacity.

Densification is the third piece. Adaptive cloning and splitting of Gaussians is triggered only by photometric gradients that remain after the split-or-deform step. This extends to the foreground the insight of RobustSplat, whose finding is that moving content fools densification into spawning floaters. That paper protects only the static background and discards everything dynamic. We instead protect the dynamic representation itself, with the aim of keeping primitive counts, and hence latency and memory, bounded.

The probabilistic lifting of PCF-Lift supplies the intuition that noisy 2D evidence should be absorbed by explicit uncertainty rather than trusted. That method models lifted features as Gaussians and down-weights noisy mask boundaries, but it is static and runs on slow volumetric ray-marching. Our adjudication plays the analogous role for motion cues, in a dynamic scene, at rasterization speed. Its source of uncertainty is geometric redundancy, not a learned covariance.

Baselines

The naive reference point is vanilla 4D Gaussian Splatting: canonical Gaussians plus a space-time feature-plane deformation decoder, with no rigidity and no tracks. It reports 34.05 dB mean PSNR on D-NeRF at 82 FPS (800×800, RTX 3090). We will rerun it on our hardware with its released settings. Matching its FPS while exceeding its PSNR is the minimum bar.

The strongest photometric-only competitor is SC-GS. It uses about 512 control points with six-degree-of-freedom transforms regularized by an as-rigid-as-possible loss and blends motion to dense Gaussians from nearest control points. It reports 43.31 dB PSNR on D-NeRF. Because it uses no 2D tracks, it is immune to tracker errors but exposed to monocular ambiguity. Its D-NeRF figure is far above the others in the file, so we will reproduce it under our evaluation protocol before treating it as the target. On D-NeRF we count matching it within reproduction noise at higher FPS as success. On real monocular data, outperforming it is the real test.

The strongest track-driven competitor is From Tokens to Nodes. It lifts DINOv2 patch tokens into 3D control nodes and drives them with splines initialized from 2D tracks, reporting 25.78 dB mean PSNR on HyperNeRF with 25 MB storage. We will feed it and our method the identical track and depth sets, so that differences reflect how the cues are used. Beating it means higher PSNR at equal or smaller storage, and above all a flatter degradation curve under track corruption.

For multi-view real scenes, MEGA is the efficiency reference: 33.57 dB PSNR and 83.14 FPS on Technicolor, versus 32.07 dB and 55.26 FPS for 4DGS. HoliGS is the hierarchy reference at ~20 FPS. No reported PSNR for HoliGS on a benchmark in our file was found on Lacuna, so we compare to it only on rendering speed and on our own reruns. DynaSplat, which mixes offset variance with optical flow and reports 34.39 dB on D-NeRF, is the closest "use 2D motion but cross-check it" competitor. It cross-checks for static/dynamic separation, not for track reliability.

Every method gets the same frames, cameras, upstream trackers, training-iteration budget, and GPU, and FPS is measured at identical resolution.

Three ablations isolate the key ingredient. In "rigidity without adjudication," all tracks get unit weight. In "adjudication without photometric check," the weight depends only on $r_j$, which reduces to a robust loss. The third replaces our discrete clusters with a learned continuous per-Gaussian rigidity weight, which is the obvious design for this intersection.

Why not just…?

Why not just put a robust loss (Huber or Cauchy) on the track term? This is the most serious threat, and it is exactly our second ablation. A robust loss down-weights large residuals regardless of cause. So on a cloth corner or a bending arm, it suppresses the true signal that the motion is non-rigid, and the reconstruction stays falsely rigid or smears. Our photometric check exists to separate "large residual because the track is wrong" from "large residual because the object bends." If the robust-loss ablation matches us on scenes with genuine deformation (D-NeRF Stand Up, Mutant; HyperNeRF Peel Banana, Chicken), the photometric routing is not earning its keep, and we would simplify the method accordingly.

Why not just trust the tracker's own visibility or confidence output? Tracker confidence is computed from the same 2D evidence that fails under occlusion and motion blur. The decoupled-pipeline literature in our file shows that downstream lifters cannot recover what the 2D stage lost. Split4D shows that tracking-based decomposition (SA4D, 0.592 mIoU) collapses relative to native optimization (0.893 mIoU) on interacting humans. We will run a variant that weights tracks by tracker confidence and compare, track by track, how often each signal flags injected errors.

Why not just drop the tracks and optimize photometrically, as SC-GS does? On synthetic D-NeRF this may work as well as anything, and we say so openly. The case for using tracks is monocular real capture. There, 4D Gaussian Splatting and Deformable 3D Gaussians both acknowledge that camera motion and object deformation are ambiguous without auxiliary priors. The HyperNeRF comparison against a reproduced SC-GS settles this. If SC-GS ties us there, our contribution narrows to robustness and speed.

Why not just learn a continuous per-Gaussian rigidity weight and skip the clustering? That is the third ablation. A per-Gaussian weight gives redundancy no grip, since a single Gaussian cannot outvote a lying track, and it reintroduces per-primitive deformation cost at render time.

Proposed Experiments

The decisive experiment is a controlled track-corruption sweep. On D-NeRF and HyperNeRF, we take the output of a standard 2D tracker and inject three kinds of corruption at rates from 0 to 40 percent: drift (gradual offset), identity swaps onto adjacent surfaces, and occlusion-time jumps. We then plot PSNR, LPIPS, and Gaussian count against the corruption rate for our method, From Tokens to Nodes, the robust-loss ablation, and the confidence-weighted variant. The hypothesis is that our curve stays nearly flat while the track-driven baseline degrades steeply. We also predict that our Gaussian count stays bounded while the others inflate through spurious densification. Because injected errors are known, we can directly measure precision and recall of the adjudication against them.

The second check is whether rigidity is discovered correctly. On D-NeRF we inspect the cluster structure per scene: Bouncing Balls should yield purely rigid clusters, T-Rex and Mutant articulated splits, and Stand Up a mix of rigid limbs and soft residual regions. Our scene-level summary statistic is the fraction of Gaussians that receive residual capacity, since it directly predicts render cost.

The third check concerns latency and memory under real multi-view capture. On Technicolor and N3DV we report PSNR, SSIM, LPIPS, FPS, and storage against MEGA and 4DGS on the same GPU. No reported N3DV PSNR for MEGA, DynaSplat, Split4D, or 4D-GS was found on Lacuna, so all N3DV numbers will be our reruns. The target is MEGA-level quality (33.57 dB on Technicolor) at or above its 83.14 FPS without its entropy-balancing sensitivity.

A final falsification test uses contact-heavy and topology-changing sequences (HyperNeRF Peel Banana, Broom). Smooth-deformation methods are known to fail on these, and the split-or-deform rule may also fail if peeling produces many small incoherent motions. We will report these failures as the method's boundary rather than tune them away.