When a 3D scene is rebuilt from video with Gaussian splatting (a representation that draws the scene as millions of small, soft, colored blobs), the optimizer adds blobs wherever the image has sharp detail it cannot yet reproduce. It cannot tell why the detail is sharp. The printed pattern on a shirt and the silhouette of the arm in front of it produce the same kind of intensity jump, so both trigger blob splitting. In dynamic scenes the cost is high. Every extra blob on a moving pattern needs its own motion, which inflates memory and destabilizes the network that predicts how blobs move over time. The real silhouettes, meanwhile, stay blurry, because soft blobs are a poor fit for hard edges. Single-image methods face the same confusion and resolve it with learned cues or iterative diffusion. Video carries a physical cue that single images lack. A painted pattern moves with its surface, so image motion is continuous across it. An occlusion boundary separates two surfaces that move differently or sit at different depths, so motion jumps across it and pixels appear or disappear.
We propose to build this cue into the rule that decides when a dynamic Gaussian model adds geometry. Where a large reconstruction error coincides with a motion or occlusion discontinuity, the model splits blobs and gives them sharp profiles. Where the error sits on motion-continuous texture, the model instead raises the texture resolution carried by the existing blob and leaves the geometry and the motion model alone. The claim is testable. On dynamic benchmarks, this "motion gate" should cut the number of primitives on textured regions, sharpen moving silhouettes, and match or beat frequency-aware dynamic splatting in image quality.
Core Claim
Current dynamic splatting treats all high-frequency image content alike. Frequency-aware Dynamic Gaussian Splatting (FAGS) splits primitives into sharp and smooth kernels and adds a Fourier-based deformation network. It correctly diagnoses that dense blob stacking at sharp edges destabilizes motion. Its sharp kernels, though, are assigned by spatial frequency alone, so a checkered shirt receives the same high-frequency treatment as the arm's outline. Static work has solved each half of the problem separately. DisC-GS shows that true silhouettes are better modeled by clipping Gaussians along curves than by stacking them, and A²TG shows that most primitives need almost no texture while the rest can carry adaptive per-primitive texture instead of subdividing. Neither has a notion of time. NEF shows that multi-view consistency filters out texture edges in 3D, but only for static, calibrated objects. Our claim is that in dynamic captures the texture-versus-boundary decision can be made cheaply and per pixel from motion discontinuities. That decision is what should route capacity between geometry (splitting and sharp or clipped kernels) and appearance (per-primitive texture). To our knowledge this routing does not exist in any dynamic method in the literature we reviewed.
Research Program
The system starts from an existing 4D Gaussian codebase, the public MEGA implementation of MEGA. Its deformation field and pruning are left untouched so that any gain can be credited to the gate. For each training frame we compute forward and backward optical flow (per-pixel image motion between consecutive frames) with an off-the-shelf estimator, as RoDynRF does. From it we derive a per-pixel boundary score:
$$ b(p) = \max\Big(\tfrac{\max_{q \in \mathcal{N}(p)} \lVert f(p) - f(q) \rVert}{\sigma_f},\; o(p),\; \tfrac{\max_{q \in \mathcal{N}(p)} |d(p) - d(q)|}{\sigma_d \, d(p)}\Big) $$
Here $f(p)$ is the flow at pixel $p$ and $\mathcal{N}(p)$ its small neighborhood. $o(p)\in[0,1]$ marks occlusion or disocclusion, detected where forward and backward flow disagree. $d$ is the model's own rendered depth, which supplies a second opinion once geometry has formed. $\sigma_f, \sigma_d$ are normalizers set from per-sequence statistics.
During densification, the usual gradient-triggered split of a Gaussian is allowed only if the pixels it projects to have mean $b$ above a threshold $\tau$. Those Gaussians may also switch to a sharp or clipped profile in the manner of DisC-GS. Below $\tau$, the same error signal instead raises that Gaussian's texture resolution, as in A²TG, with shape unchanged and no new motion parameters.
The analysis part of the program asks where the cue breaks. Three regimes need direct study. The first is a static camera watching a static object edge, where flow carries no information and only the depth term or the multi-view signal on multi-camera rigs can help. The second is moving cast shadows, which create flow discontinuities on flat ground; here we expect the depth term to veto splitting, and we will measure whether it does. The third is non-rigid cloth wrinkles, which are real but shallow geometry that flow may call texture.
Baselines
The naive reference is the unmodified base 4D Gaussian pipeline (MEGA as released), trained with its default schedule. It shows how much primitive growth and edge blur the gate removes. On Technicolor, MEGA reports 33.57 dB PSNR with 0.91M Gaussians on "Birthday", against 32.07 dB for original 4DGS.
The strongest competitor on this question is FAGS. It is the only method in the file that explicitly allocates sharp versus smooth primitives in dynamic scenes, which is the exact decision our gate makes differently. It reports average PSNR 42.76 on the D-NeRF synthetic benchmark and 32.18 on Neu3D, versus 41.99 and 31.63 for Grid4D (FAGS). No FAGS code is listed in the file. If none is available, we will reimplement its frequency-differentiated kernels and FFT loss on the same base codebase and check that we reproduce its reported numbers within 0.3 dB before comparing. All methods get the same frames, camera poses, iteration count, and a matched hyperparameter sweep of equal size. Beating FAGS means matching its PSNR within noise while using fewer primitives, with higher silhouette accuracy and lower error on textured regions. A PSNR gain alone would not count.
Two ablations isolate the key ingredient. The first is "texture capacity without the gate": adaptive per-primitive texture added everywhere, with ordinary densification. The second is "depth-only gate": $b$ computed from rendered depth alone, with no flow term. A third variant, "flow-only gate", measures the depth veto on shadows.
Why not just…?
Why not just give every dynamic Gaussian adaptive texture and let optimization sort it out? This is the most serious threat. A²TG shows that 62% of static Gaussians need only a 1×1 texture, so capacity alone may absorb textures. Standard densification still fires on positional gradients, however, which texture does not suppress. The "texture without gate" ablation settles the question directly. If it matches the full method on primitive count and silhouette accuracy, the gate is unnecessary and we will report that.
Why not just use rendered depth discontinuities, which need no motion? Early in training the rendered depth is blurry. That is exactly when densification decides where blobs go, and it is the same early-stage fragility that 3DGS-HPC documents for features computed from half-formed renders. Externally computed flow is sharp from the first iteration. The depth-only ablation, with $b$ logged over training, will show whether the flow term matters early and becomes redundant late.
Why not just segment objects with SAM or a video tracker and split only at mask edges? Semantic masks cover whole objects, so they fire on part boundaries and logos. They also miss non-semantic movers such as shadows and dragged items, per 3DGS-HPC. Tracking-based decomposition reaches only 0.592 mIoU against Split4D's 0.893 on the multi-human benchmark (Split4D). We will include a mask-edge gate as an extra baseline on the retextured set described below, where logos make its failure measurable.
Why not just prune afterwards, as MEGA does with opacity entropy? Pruning removes redundant blobs after the fact. It does not stop texture blobs from receiving motion parameters during training, and it does not sharpen silhouettes. Comparing against MEGA's pruning on both primitive count and edge accuracy tests exactly this.
Proposed Experiments
We evaluate on the D-NeRF synthetic benchmark (monocular with a moving camera, so parallax helps the cue), Neu3D, and Technicolor (static multi-camera rigs, the hard case for flow). Metrics are PSNR, SSIM, LPIPS, Gaussian count, and storage, plus two targeted measures. The first is image error restricted to textured-flat pixels versus silhouette pixels. The second is a boundary F-score computed against ground-truth depth discontinuities, which the synthetic scenes provide.
The decisive check is a controlled retexturing study. We re-render D-NeRF synthetic sequences with identical geometry and motion but increasingly aggressive surface patterns: plain, then checkerboard, then printed text. If the D-NeRF assets cannot be retextured, we will build comparable scenes. The prediction is that baselines' primitive counts grow with texture contrast while ours stays nearly flat, and that silhouette F-score does not drop. If our count grows as fast as FAGS's, the core claim fails.
A second check targets the shadow failure mode. On the Neu3D sequences with visible moving shadows, we manually annotate shadow regions and measure false splits there with and without the depth veto. Finally, on Technicolor we report results stratified by pixel motion to show where the gate adds nothing. Static-rig, static-object regions should behave like the baseline, which bounds the claim honestly.
References
- Frequency-aware Dynamic Gaussian Splatting
- DisC-GS: Discontinuity-aware Gaussian Splatting
- A$^2$TG: Adaptive Anisotropic Textured Gaussians for Efficient 3D Scene Representation
- NEF: Neural Edge Fields for 3D Parametric Curve Reconstruction from Multi-View Images
- MEGA: Memory-Efficient 4D Gaussian Splatting for Dynamic Scenes
- Robust Dynamic Radiance Fields
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation