Rendering a moving scene in real time forces a choice today. One option is to deform every 3D primitive with a neural network, which captures cloth folds and skin stretch but is slow. The other is to move primitives in rigid clusters, which is fast but blurs anything that bends. Separately, a scene can be kept editable by splitting it into objects, but current object-level methods assume each object moves rigidly. We propose one representation that serves all three goals. The scene is split into object slots. Each slot moves by smoothly blending a handful of control points. A per-primitive switch, learned during training, turns on an expensive non-rigid correction only for the few primitives that actually need it. Grouping primitives by shared motion is what makes an object editable, and it is also what makes rendering cheap, so the two problems can be solved by the same structure.
Concretely, the base representation is 3D Gaussian splatting: the scene is a cloud of colored, semi-transparent ellipsoids drawn by fast rasterization. Our claim is that the cost of non-rigid fidelity should scale with the fraction of the scene that is truly non-rigid, not with the total number of Gaussians. A bookshelf costs nothing, a rolling ball costs one rigid transform, and a person's sleeve costs per-Gaussian corrections, all inside a single scene whose objects can be moved independently afterward.
Core Claim
Existing work splits along the axes this proposal joins. Superpoint Gaussian Splatting groups Gaussians into rigidly moving "superpoints" and reaches up to 227 FPS on synthetic data. Its authors report two costs. Hard rigid clusters create seams and lose PSNR against per-Gaussian deformation. Their non-rigid add-on (SP-GS+NG) recovers detail but drops speed to about 119 FPS, because the correction network runs on every Gaussian. Their superpoints are also not editable objects.
R5DGS attaches identity vectors to Gaussians and propagates rigid motion from group centroids for speed. It explicitly loses fidelity in deformable regions. Split4D decomposes scenes without a video tracker, but its objects move with linear velocities, and boundaries blur under close contact. Neural Parametric Gaussians handles a single non-rigid object well but does not separate it from a background.
The gap is a decomposition whose objects are allowed to deform non-rigidly, together with a motion model whose run-time cost tracks how much non-rigid motion there actually is. We claim three things can be had at once: rendering speed near rigid-cluster methods, fidelity near per-Gaussian deformation, and object-level counterfactual editing (moving or retiming an object along a trajectory it never followed in the video). The ingredient that makes this possible is a sparse, learned residual gate.
Research Program
Each Gaussian $i$ carries a soft assignment to one of $S$ object slots, learned from a feature vector as in Split4D and R5DGS. Each slot $s$ owns a small set of control nodes. Each node has a time-varying rigid transform $T_{j}(t)$, stored as explicit keyframes with spline interpolation in the manner of Ex4DGS, so evaluating a node costs no network call.
A Gaussian at canonical position $\mathbf{x}_i$ moves as
$$\mathbf{x}_i(t) = \sum_{j \in \mathcal{N}_s(i)} w_{ij}\, T_j(t)\,\mathbf{x}_i \;+\; g_i\, \Delta_\theta(\mathbf{x}_i, t).$$
Here $\mathcal{N}_s(i)$ is the set of nearest control nodes belonging to the Gaussian's own slot $s$, $w_{ij}$ are smooth blending weights, $\Delta_\theta$ is a small shared network that outputs a per-Gaussian displacement, and $g_i \in [0,1]$ is a learned gate.
Restricting blending to nodes of the same slot matters in two ways. Inside an object, blending removes the seams of hard superpoints. Between two objects, the restriction keeps a hand and the cup it holds from smearing into each other, which is exactly the contact failure reported for Split4D. Static background is a slot with zero nodes. Nodes in a slot whose trajectories agree are merged during training, so a rigid object ends up with a single node.
The gate is trained with a cost-weighted sparsity penalty added to the usual photometric loss $\mathcal{L}_{\text{photo}}$:
$$\mathcal{L} = \mathcal{L}_{\text{photo}} + \lambda \sum_i c\, |g_i| + \mu\, \mathcal{L}_{\text{slot}}.$$
The constant $c$ is the measured per-Gaussian cost of one evaluation of $\Delta_\theta$, and $\lambda$ trades frame time against image quality. At render time $\Delta_\theta$ runs only on Gaussians with $g_i > \tau$. Per-frame cost is then roughly a fixed rasterization cost, plus a cheap blend over all $N$ Gaussians, plus $\rho N$ network evaluations, where $\rho$ is the fraction of gates that are open. Speed should therefore approach plain SP-GS as $\rho \to 0$ and approach SP-GS+NG as $\rho \to 1$. Sweeping $\lambda$ traces a quality-versus-speed curve, and that curve is the main object we report.
The slot term $\mathcal{L}_{\text{slot}}$ asks Gaussians in the same slot to share both node motion and 2D foundation-model features. The features are lifted into 3D following the distillation idea of Dumery et al.. Motion alone cannot separate objects that never move, which is HEIR's "static problem" (HEIR), so the appearance features are what keep stationary objects apart.
We assume calibrated cameras throughout and leave pose estimation without SfM out of scope. This is deliberate: the first question is whether sparse residuals buy the claimed trade-off. For the same reason we do not attempt relighting or shadow removal. Shadow artifacts that appear on object removal are reported as a known failure mode, not solved.
Baselines
The naive reference is static 3DGS trained on all frames. It shows how much ghosting the static assumption costs. D-NeRF shows the scale of that cost: on Mutant, a static NeRF reaches 20.31 dB against 31.29 dB for the deformation model.
The strongest speed-fidelity competitor is SP-GS together with its SP-GS+NG variant. It occupies exactly the trade-off we claim to improve: up to 227 FPS for rigid superpoints and about 119 FPS with non-rigid refinement on a V100. It has no reported per-scene PSNR in our file, so we will rerun its released code ourselves. The strongest monolithic real-scene methods are FRoG, which reports 31.01 dB PSNR, SSIM 0.941 and 91.13 FPS on N3DV, and Ex4DGS, which reports 32.11 dB at 62.1 FPS on a 2080Ti. For decomposition, the strongest competitor is Split4D, which reports mIoU 0.893 on Multi-Human against 0.592 for SA4D. R5DGS represents rigid-object decomposition.
Fairness rules apply to every method. All methods use the same initial point cloud and camera poses, the same training-iteration budget, and the same GPU; we time on both a V100 and a consumer card. Each method gets an equal hyperparameter search of a fixed number of trials on a held-out view.
Beating SP-GS means one of two things on D-NeRF, HyperNeRF and N3DV:
- matching SP-GS+NG's PSNR and LPIPS at clearly higher FPS (for example, within 0.2 dB at more than 1.5× its frame rate), or
- matching rigid SP-GS's frame rate at clearly higher PSNR.
Beating FRoG on N3DV means exceeding 31.01 dB without dropping below its 91 FPS on matched hardware. If we cannot match its PSNR, we must at least add editing that FRoG cannot do.
Two ablations isolate the key ingredient. Gate-always-on recovers a skinning-plus-full-residual model, which tests whether sparsity costs fidelity. Gate-always-off recovers pure blended skinning, which tests whether the residual is needed at all. A third ablation removes the same-slot restriction on blending, to show that the restriction is what keeps contact boundaries sharp.
Why not just…?
Why not just run SP-GS+NG, or FRoG, and accept roughly 100 FPS? This is a real threat. If image quality and speed on today's benchmarks were the only goals, these methods may already be good enough. Our answer has two parts.
The first part is scaling. Their cost grows with the total number of Gaussians, so it worsens in multi-object scenes, at headset resolutions, and on weak hardware. The latency motivation from VR avatars in ESCA points to exactly that regime. We will measure FPS as the Gaussian count grows while the non-rigid fraction stays fixed. Our cost should grow more slowly.
The second part is editability. Neither method produces objects that can be moved. If our speed curve does not beat SP-GS+NG on the D-NeRF and N3DV scenes, the claim reduces to editability alone, and we will report it that way.
Why not just train a monolithic deformable model and cluster its Gaussian trajectories afterward? Ex4DGS already separates static from dynamic Gaussians by thresholding motion residuals, so clustering after training is plausible. It still leaves two things missing. Motion would live in world coordinates rather than object-local frames. And there would be no guarantee that clusters respect contact. The counterfactual benchmark below tests this directly. We will include a variant that clusters per-Gaussian trajectories after training, then retimes the clusters, and compare it against our slots.
Why not just mask each actor with an off-the-shelf 2D segmenter and fit a separate deformable model per object? Pollefeys and colleagues (Robust Dense Mapping) show that discarding dynamic regions leaves a system blind to them. Split4D and R5DGS both report that mask trackers swap identities and fail under occlusion and close contact. We will still run this baseline, using SAM2-style masks, on Multi-Human and SelfCap and on our contact scenes. If it matches us on mIoU and novel-view PSNR, our joint optimization is not justified.
Why not just use rigid boxes per actor? Lindström et al. show that rigid boxes fail on articulated people. Our gate-always-off ablation, with a single node per slot, is this baseline, so it is covered directly.
Proposed Experiments
The speed-fidelity experiment sweeps $\lambda$ on D-NeRF, HyperNeRF, N3DV and NeRF-DS. For each setting we plot PSNR, SSIM and LPIPS against FPS, and also report the open-gate fraction $\rho$. The decisive check is whether our curve dominates the line joining SP-GS and SP-GS+NG. A second check is whether $\rho$ stays small, under about 20%, at near-full fidelity. If the gates open everywhere, the premise that non-rigid motion is spatially sparse is false and the proposal fails. We will also visualize the open gates. They should concentrate on cloth, faces and hands rather than on background or rigid parts. We cross-check rigid-part behavior on D-NeRF Excavator against HEIR's reported 21.56 dB, which compares with 19.91 dB for SC-GS.
The decomposition experiment reports mIoU on Multi-Human and SelfCap against Split4D's 0.893. We also add a boundary-band PSNR, computed only within a few pixels of contact edges, to test the same-slot blending claim.
The decisive new test is a counterfactual-animation benchmark built from DeformingThings4D and D-NeRF assets. Two or more non-rigid assets are composited into a static room and rendered from multiple views. Models train on one choreography and are evaluated, against ground-truth renders, on a different one. The test choreography relocates one asset and offsets it in time. No monolithic method can attempt this without our post-hoc clustering variant. We report PSNR and LPIPS on the edited renders, along with leakage, defined as the photometric change in the untouched object.
A secondary, monocular stress test on DyCheck compares against OriGS's reported 19.69 dB with ground-truth poses. Here we only expect to show whether the slot structure degrades gracefully, because under-constrained monocular deformation is a known failure mode for template-free models, as noted for NPG.
References
- Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction
- R5DGS: Semantic-Aware 4D Gaussian Splatting with Rigid Body Constraints for Efficient Dynamic Scene Reconstruction
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction
- Fully Explicit Dynamic Gaussian Splatting
- Enforcing View-Consistency in Class-Agnostic 3D Segmentation Fields
- HEIR: Learning Graph-Based Motion Hierarchies
- D-NeRF: Neural Radiance Fields for Dynamic Scenes
- Fast and Robust Deformable 3D Gaussian Splatting
- ESCA: Enabling Seamless Codec Avatar Execution through Algorithm and Hardware Co-Optimization for Virtual Reality
- Robust Dense Mapping for Large-Scale Dynamic Environments
- Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap