Rank-Zero Means Static: One Per-Frame Motion Budget for Fast, Mask-Free Dynamic Splatting

Dynamic scenes break the basic contract of 3D reconstruction, which is that each point in space holds the same matter from every view and at every moment. Current fixes split motion into a large rigid movement plus small local wiggles, or split the scene into a static background and moving foreground. Each fix usually adds a neural network that must run for every rendered primitive in every frame, and that cost erodes the real-time speed that made Gaussian splatting attractive. Gaussian splatting represents a scene as millions of small colored 3D blobs that are projected onto the image. We propose a single representation that does both splits at once. Each blob carries a small "motion budget." A blob with zero budget is static background. A blob with budget one rides the rigid trajectory of one discovered moving entity. A blob with a larger budget also adds a few shared non-rigid residual motions, expressed in that entity's local coordinate frame. All time-dependent quantities are evaluated once per frame per entity, never per blob, so the rendering cost stays close to that of static splatting.

The core bet is that a hierarchical factorization does three jobs at once. It is the motion-deformation disentanglement that lets optimization survive large translations. It is the static/dynamic decomposition, discovered without segmentation masks. It is also the latency mechanism, because the work that depends on time shrinks from the number of blobs to the number of entities plus the number of motion bases. What is new is having all three in one structure, applied to scenes with several moving entities, large rigid motion, and a real background. Existing methods each cover only one or two of these.

What exists and where it stops

Three systems sit closest to this idea, and each covers a different two-thirds of it. Motion Decoupled 3D Gaussian Splatting (M5D-GS) separates an object-level rigid 6-degree-of-freedom transform, predicted by a small network, from per-blob deformation predicted by a second network. It survives large translations by inflating blob sizes early in training, so that rendered blobs still overlap the observed object and gradients stay informative. We reuse that warm-up. M5D-GS is limited to single isolated objects, and its per-blob deformation network runs at render time.

DynMF supplies our latency mechanism. It expresses each blob's trajectory as a sparse mix of 10–32 shared basis trajectories, produced by a tiny network queried once per frame. Its bases are world-space displacements, and it does not separate entities from background or rigid motion from articulation.

Fully Explicit Dynamic Gaussian Splatting (Ex4DGS) supplies the idea that every blob starts static and is promoted to dynamic when its error stays high. It also shows that trajectories can be stored as splines with no network at all. Its splines are per blob, however, so a whole actor translating across a room must be stored blob by blob, with no kinematic structure.

Other work motivates the problem without solving it. Total-Recon shows that separating root-body motion from articulation stabilizes long sequences: it reports 0.278 LPIPS versus 0.611 for D2NeRF. It relies on Mask R-CNN masks and needs about 15 hours on four GPUs per sequence. Split4D discovers entities without video segmentation, reaching 0.893 mIoU on Multi-Human versus 0.592 for the tracker-based SA4D. Its per-blob linear velocities have no rigid/non-rigid hierarchy, and it reports blurring where entities touch.

Two papers constrain the design. 4D Gaussian Splatting and M5D-GS's large-motion experiments show that a single monolithic deformation field fails under large rigid translation. STDR shows that letting static and dynamic blobs share one canonical space causes ghosting.

Core Claim

The position of blob $i$ at time $t$ is

$$ \mathbf{x}_i(t) = G_{k(i)}(t)\Big(\mathbf{x}_i + \sum_{b=1}^{B} w_{ib}\,\mathbf{r}_{k(i),b}(t)\Big). $$

Here $\mathbf{x}_i$ is the blob's canonical position, and $k(i)$ is the entity it belongs to. Entity $0$ is the static background, and its transform $G_0$ is the identity. $G_k(t)$ is entity $k$'s rigid rotation and translation at time $t$. $\mathbf{r}_{k,b}(t)$ is the $b$-th residual displacement basis, expressed in entity $k$'s local frame. The weights $w_{ib}$ are sparse per-blob mixing coefficients. Blob rotations compose the same way.

A blob's motion budget is the number of nonzero terms it uses. Zero means static. One means purely rigid. Higher values add residual deformation. Group-sparsity penalties push every blob toward the smallest budget that explains its pixels.

Per frame, the renderer evaluates $K$ rigid poses and $KB$ basis vectors, either from splines or from one tiny network call. Each blob then needs one transform and at most a couple of additions. Static blobs skip the motion step entirely. We claim this matches or beats current methods on scenes with large motion and several entities, while rendering within a small factor of static splatting. We also claim the hierarchy, not the low-rank bases alone, is what makes this work.

Research Program

We would build the representation on standard splatting, as in 3D Gaussian Splatting. Entity membership starts as a soft assignment over a fixed number of slots plus the background slot. An entropy penalty anneals it toward a hard assignment, so that at render time each blob reads exactly one rigid transform.

Entities are created as in Ex4DGS. Every blob begins in the background slot. Blobs whose motion residuals stay high are clustered by trajectory consistency into a new slot. A slot is split when a single rigid transform cannot explain its members' residual motion with a few bases.

Each new slot gets its own instance of M5D-GS's blob-inflation warm-up, which protects it against large initial translation errors. Adaptive densification, the step where splatting adds blobs in high-error regions, is gated by motion budget. Error at budget-zero blobs that coincides in time with a moving entity counts as evidence for promotion, not as missing background detail. This targets the mid-air artifacts described in RobustSplat.

After training, we bake the entity poses and bases into splines, following Ex4DGS's position and rotation interpolation, so that rendering involves no network. Storage then grows with entities times bases times keyframes, not with blobs times keyframes.

Analysis has three parts. We measure how motion budgets distribute across scene regions. We measure whether discovered entities match ground-truth objects. We profile where frame time goes: sorting, rasterization, or motion evaluation.

Baselines

The naive reference point is static 3D Gaussian splatting trained on the whole sequence, which exposes the floaters and blur from violating the static assumption. D-NeRF shows the scale of that failure on Mutant: 20.31 dB for static NeRF versus 31.29 dB for D-NeRF.

DynMF is the strongest competitor on the latency and factorization question. It already evaluates motion once per frame and reports 36.89 PSNR on D-NeRF at 120–300 FPS. We would run it with its released configuration. In a second configuration, we match its total number of bases to our $KB$ and give it the same blob-inflation warm-up. This ensures any gap is not just the warm-up.

M5D-GS is the strongest competitor on large motion, reporting 35.86 PSNR on Jumping Jacks + Motion at 135.15 FPS. On multi-entity scenes we run it once per entity, using SAM-style masks, and composite the results over a static splatting background. That is the best fair use of a single-object method.

Ex4DGS is the strongest network-free and static/dynamic baseline, reporting 32.11 dB PSNR on N3V with sparse initialization at 62.1 FPS on an RTX 2080 Ti. On full-quality D-NeRF, SC-GS reports 43.31 PSNR, which is the number to approach where large motion is absent.

For decomposition quality, Split4D's 0.893 mIoU on Multi-Human is the reference. All methods are retrained by us on the same frames, the same GPU, the same resolution, and with equal tuning trials. We report the published numbers alongside our reruns.

Beating the baselines means two things. On the multi-entity large-motion benchmark, we need higher PSNR than every baseline at no worse than 0.8× DynMF's frame rate. On standard benchmarks, we need to stay within 1 dB of the best published number.

The key ablation removes the entity rigid layer by setting every $G_k$ to the identity. With budget-zero static blobs kept, this reduces the model to DynMF. It isolates whether hierarchy, rather than sparsity, carries the gain. A second ablation replaces the shared bases with M5D-GS's per-blob deformation network, to measure the latency cost the bases avoid.

Why not just…?

Why not just use DynMF with more bases? This is the real threat, and we take it seriously. The displacement of a rigidly moving point is linear in its canonical position with time-varying coefficients. DynMF's per-blob coefficients can therefore represent rigid motion in principle. Our argument concerns optimization and structure, not expressiveness. Nothing ties those coefficients to position, so under large translations the same overlap starvation that M5D-GS analyzes should apply. Without an entity layer, there is also no mask-free grouping into objects and no meaningful rigid pose to edit. The matched-basis, matched-warm-up DynMF run settles this. If it comes within 0.3 dB of our method on the composite benchmark and its coefficient clusters match ground-truth entities, the hierarchy is unnecessary. We would then report the warm-up plus static rank-zero gating as the contribution instead.

Why not just run per-blob splines as in Ex4DGS? Ex4DGS's spline storage grows with the number of dynamic blobs. Under an actor-wide translation, every blob of that actor must be promoted. Its interpolation also has no kinematic structure, so it cannot cleanly move one entity while freezing the rest. Our storage and editing comparison on the composite scenes measures exactly this. Gap 3 of the file states the trade-off. Its "Cook Spinach" result of 115 MB comes from a scene with little rigid motion, so it is not the regime we target.

Why not just segment with SAM and run M5D-GS per object? Split4D's comparison against the tracker-based SA4D (0.592 versus 0.893 mIoU) and Total-Recon's own ghosting failures show that 2D mask errors propagate into 3D. Running one model per object also multiplies per-blob network cost by the number of entities. We include this pipeline as a baseline rather than dismissing it. If mask quality on our scenes is high, it may match us on image quality. In that case our remaining claim would rest on speed and on contact-zone boundaries.

Why not distill M5D-GS's networks into per-frame lookup tables after training? This removes runtime network cost, but storage then grows as blobs times frames. It also inherits the single-object limit. We report it as a "bake-after" ablation measuring both storage and frame rate.

Proposed Experiments

The decisive benchmark is one we must construct. We will composite two to four objects from M5D-GS's augmented large-motion D-NeRF scenarios, each with independent large rigid trajectories, over static backgrounds taken from N3V-style captures. Ground-truth entity masks and poses come for free. On this benchmark we report PSNR, SSIM, LPIPS, frames per second at fixed resolution, storage, and entity mIoU. Background-only PSNR is computed on regions that moving entities have vacated, which directly measures leftover static-assumption artifacts.

Secondary checks cover four existing benchmarks. On standard D-NeRF, the method should not lose much where the hierarchy is unneeded. On N3V we compare against Ex4DGS on real multi-view footage. On Multi-Human we compare against Split4D's 0.893 mIoU. On Total-Recon's Stereo RGBD sequences we test long durations: no rendering speed is reported on Lacuna for Total-Recon, but its quality figures give a target.

The experiment that would most change our view is the hierarchy ablation on the composite benchmark. If removing $G_k$ costs less than the gap between DynMF and our method's warm-up variant, the motion budget is not doing the work we claim. A frame-time breakdown showing motion evaluation under 10% of render time would confirm the latency argument. A larger fraction would mean the hard assignment is not delivering the per-entity cost we expect.