Motion-Tolerant Losses for Sharp Dynamic Gaussian Splatting

When a model reconstructing a moving scene is unsure where a hand or a wheel is at a given instant, the standard per-pixel loss punishes a sharp hand drawn in slightly the wrong place twice. It is wrong once where the hand should be and once where it was drawn. A blurry, half-transparent smear spread over all plausible positions is punished less, so the model learns the smear. Weather forecasters call this the "double penalty." Super-resolution researchers call the same effect regression to the mean: under uncertainty, squared-error training predicts the average of all plausible answers.

We propose to replace the per-pixel loss in dynamic 3D Gaussian Splatting with a loss that first lets the rendered image shift locally to match the observation, charging a geometric price for that shift. Dynamic 3D Gaussian Splatting represents a scene as many small colored blobs whose positions change over time. Under the new loss, a sharp but misplaced edge costs roughly "how far it is misplaced," not "twice its contrast." The shift the loss discovers is then used as a motion signal for the moving blobs, instead of being read as a request to add more blobs or lower their opacity. The claim is that much of the motion blur, ghosting and floater formation in dynamic reconstruction comes from this mismatch between a photometric loss and a geometric uncertainty. Fixing the loss should fix a large part of these artifacts without a generative model.

Core Claim

Two literatures describe the same failure from opposite sides. In the first, EnhanceNet shows that squared-error training yields blurry images because it predicts the conditional mean of many plausible sharp images. Its remedy is perceptual, texture and adversarial losses. In the second, reconstruction methods that assume a static scene break when things move. NeRF-style models resolve conflicting observations of a moving object with semi-transparent ghosts. Gaussian Splatting treats the error spikes caused by moving objects as missing geometry, and its densification step (which splits or clones blobs wherever the error gradient is large) creates floaters, as documented by RobustSplat.

Our reading is that both are one effect. The uncertain quantity in a dynamic scene is mostly a displacement: where a surface is at time $t$. A per-pixel loss can only express that uncertainty as intensity averaging. The specific claim is that a loss which measures local misalignment geometrically will do three things on monocular dynamic benchmarks, where position ambiguity is largest. It will produce visibly sharper moving regions and fewer floaters, without hallucinating content. It will also send its gradients through the motion model rather than through opacity and densification.

Concretely, for a rendered frame $R$ and observed frame $I$, the loss is

$$ \mathcal{L}_{\text{mt}}(R, I) = \sum_p \min_{u_p \in W} \Big( \lVert I(p) - R(p + u_p) \rVert_1 + \lambda \lVert u_p \rVert^2 \Big) + \mu\, \mathcal{S}(u), $$

where $p$ indexes pixels and $u_p$ is a local displacement restricted to a small window $W$. The weight $\lambda$ sets the price per squared pixel of shift. The term $\mathcal{S}(u)$ is a smoothness penalty on the displacement field, weighted by $\mu$. As $\lambda \to \infty$ the loss reduces exactly to the standard $L_1$ loss. This gives a clean one-knob ablation from "today's training" to "fully motion-tolerant."

The intuition for why sharpness wins can be checked numerically. Take an edge of contrast $c$ whose true position is spread with standard deviation $\sigma$. Rendering it sharp at the most likely position costs about $\lambda \sigma^2$ per edge pixel. Rendering the blurred average costs a photometric residual that grows with $c$ and with the blur width. So sharp renders are preferred whenever $\lambda \sigma^2$ is small relative to the contrast. This scaling follows directly from comparing the two terms of the loss above.

How This Differs From the Closest Work

Four papers already sit at this intersection. Each makes a different choice from ours.

Frequency-aware Dynamic Gaussian Splatting (FAGS) is the closest. It diagnoses a "spectral conflict": dense stacks of smooth Gaussians cannot be moved coherently, so the motion model regresses to smooth trajectories. FAGS fixes this through the representation (sharp versus soft kernels, and a Fourier-embedded deformation network) plus an FFT amplitude loss. That amplitude loss is invariant only to global translation and discards phase. Its color term remains pixel-aligned $L_1$. Our proposal leaves the representation alone and changes what a local misalignment costs and where its gradient goes. The two ideas are complementary, and we test them stacked.

VDEGaussian attacks under-sampled motion by distilling a scene-adapted 2D video diffusion model into 4D Gaussians, gated by a learned uncertainty map. It gains sharpness by sampling plausible content, and it pays with 3D inconsistency and per-scene adaptation cost. We do not hallucinate. We only refuse to average over positions the data cannot pin down.

BARD-GS models physical exposure blur by averaging renders along virtual camera poses and intra-exposure Gaussian motion, guided by external tracks. It addresses blur present in the input. We address blur created by the training objective even from sharp inputs, a failure FAGS demonstrates on synthetic data.

SRL shows, for 2D slot-based video models, that blurry decoders trained with squared error feed weak gradients back to the encoder that groups objects. This motivates our gradient-routing step: in 4D Gaussians, the analogous loop is blurry renders feeding diffuse gradients to the deformation field.

Research Program

The method has three pieces, built on the open deformation-field model 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (4D-GS). That model attaches a time-dependent offset network to a canonical set of Gaussians.

The loss. The minimization over $u_p$ is computed with a soft-min over a $k \times k$ window of candidate shifts (we start with $k=7$). It runs on a two-level image pyramid so larger motions are captured cheaply. This keeps the loss differentiable in $R$ and costs about $k^2$ shifted residual evaluations per level. The weights $\lambda$ and $\mu$ are annealed: tolerant early, when motion is poorly known, and tighter late, so static, well-observed regions converge to pixel-accurate color. This matches the observation in 3DGS-HPC that image features computed on early, noisy Gaussian renders are fragile. Our matching is purely photometric and local, which we expect to be more robust than learned features at that stage. The experiments must confirm this.

Gradient routing. The recovered displacement field $u$ is turned into a target for the screen-space positions of Gaussians whose deformation is active. Their projected centers are pulled by $-u$ through the deformation network. Gaussians in static regions, identified by low deformation magnitude, do not receive this term. The motion signal then lands on the motion model, rather than only diffusing through color and opacity gradients.

Densification gate. Densification is triggered by the post-alignment residual instead of the raw residual. A region whose error disappears once a small shift is allowed is a motion error, so it does not spawn new Gaussians. Only error that survives alignment counts as missing geometry. This is our answer to the dilemma RobustSplat describes. Delaying densification prevents floaters but over-smooths; allowing it creates floaters. We replace the timing heuristic with a test of whether the error is explained by motion.

The analysis side is a controlled study of the double penalty itself. On D-NeRF synthetic scenes we inject known position ambiguity by jittering training-frame timestamps by controlled amounts. We then measure how the blur in moving regions grows with jitter under $L_1$ versus the motion-tolerant loss. If the mechanism is right, $L_1$ blur should grow steadily with jitter while ours stays flat up to roughly the window size.

Baselines

The naive reference is 4D-GS trained with its standard photometric loss, which is the backbone we modify. On the D-NeRF synthetic benchmark its authors report 34.05 dB average PSNR, and BARD-GS reports 22.06 dB for it on the real blurry dual-GoPro benchmark. Every variant of ours is trained from the same code, the same initialization, the same iteration count and the same densification schedule. Any difference then comes from the loss, the routing and the gate.

The strongest existing method on the question of motion blur from sharp inputs is FAGS. It reports 42.76 dB average PSNR on D-NeRF synthetic and 32.18 dB on Neu3D, above Grid4D at 41.99 dB and 31.63 dB respectively. If FAGS code is available, we run it under our hardware and iteration budget, both alone and with our loss substituted for its $L_1$ term. If only reported numbers exist, we compare against those numbers and say so. Beating FAGS on PSNR is not our primary target, because a loss that tolerates small misplacement may trade a little PSNR for sharpness. Success means two things on dynamic-region masks. First, our method on 4D-GS matches or beats FAGS in LPIPS and in a high-frequency energy ratio (rendered versus ground-truth spectral energy above a cutoff), within 0.3 dB PSNR. Second, FAGS plus our loss improves on FAGS alone in those same masked metrics.

For floaters and densification, the reference is RobustSplat's delayed densification. We reimplement its schedule on the same backbone. No reported numbers were found on Lacuna for its RobustNeRF or On-the-go averages; the file only says it leads SpotLessSplats, WildGaussians and T-3DGS. For context, 3DGS-HPC reports 1.39 to 5.54 dB gains over plain 3DGS on RobustNeRF. Beating RobustSplat means fewer Gaussians in empty space and equal or better PSNR on static content, without its delay schedule.

For blurry inputs, the reference is BARD-GS at 25.13 dB on its own benchmark. We do not expect our loss to replace its exposure model. The test is whether adding our loss inside BARD-GS's averaging improves sharpness on objects with non-constant intra-exposure velocity, the case its constant-velocity assumption handles worst.

The ablations isolate the key ingredient. One sweeps $\lambda$ from very large (equivalent to $L_1$) to small. Others remove gradient routing and remove the densification gate. A last one swaps our in-loop photometric matching for an external pretrained flow network applied between render and observation.

Why not just…?

Why not just add a perceptual loss such as LPIPS? This is the most serious competitor, because deep-feature losses are already partly tolerant of small shifts. The evidence against relying on it is indirect. EnhanceNet reports that perceptual and adversarial losses distort sensitive structures such as faces and flicker from frame to frame, and 3DGS-HPC finds deep features unreliable on early Gaussian renders. Neither result was measured on dynamic splatting under our conditions. We therefore run 4D-GS plus LPIPS at tuned weight as a full baseline, scoring temporal flicker (warping error between consecutive rendered frames) alongside sharpness. If LPIPS matches us on sharpness and flicker, the routing and gating pieces become the contribution, and we would report that plainly.

Why not just supervise motion with an off-the-shelf optical-flow or point-tracking model? BARD-GS does this with TAPIR tracks and monocular depth. Split4D shows that external 2D tracking errors propagate permanently into 4D reconstructions: its own segmentation reaches 0.893 mIoU on Multi-Human against 0.592 for a tracking-based pipeline. Our ablation that computes the displacement field with a pretrained flow network between render and observation is exactly this alternative, run inside the loop. If it matches the in-loop soft-min, the cheaper option wins and our loss reduces to "use flow between render and target." That would still be a clean, useful finding.

Why not just train coarse-to-fine, comparing blurred images early? Blurring both images enlarges the basin in which motion can be corrected. At full resolution, though, the final objective is still pixel-aligned, so wherever position remains genuinely uncertain at convergence the optimum is still the mean. We include a pyramid-annealed $L_1$ baseline with the same schedule as our annealing. The timestamp-jitter study separates the two: coarse-to-fine should help with optimization difficulty but not with irreducible ambiguity.

Why not just use FAGS's FFT amplitude loss? A global amplitude spectrum ignores where content is. It rewards the right amount of edge energy but not edges attached to the right moving object, and it gives no displacement signal to route to the deformation field. Comparing FAGS alone with FAGS plus our loss, and 4D-GS plus FFT loss with 4D-GS plus our loss, settles this directly.

Proposed Experiments

The main evaluation covers four settings. D-NeRF synthetic and HyperNeRF are monocular, where position ambiguity is highest; no reported HyperNeRF number for our baselines was found on Lacuna, so we report our own reproductions. Neu3D is multi-view and serves as a control: ambiguity is low there, and we predict small gains. The BARD-GS real blurry benchmark tests interaction with physical blur. RobustNeRF and NeRF On-the-go test the densification gate on static scenes with transient distractors.

Metrics are PSNR, SSIM and LPIPS on full frames and on dynamic-region masks, plus four further measures. The high-frequency energy ratio tests the sharpness claim. Count and total opacity of Gaussians lying outside the ground-truth or visual-hull volume tests the floater claim. Consecutive-frame warping error measures flicker. The fraction of gradient magnitude reaching deformation parameters versus opacity and densification tests the routing claim.

The decisive checks are three. First, the timestamp-jitter curve on D-NeRF must show $L_1$ blur rising with injected ambiguity while ours stays flat; if both rise together, the double-penalty account is wrong. Second, the gain must shrink from monocular to multi-view data, which a generic sharpening trick would not predict. Third, on RobustNeRF the post-alignment densification gate must reduce floaters at least as well as RobustSplat's delay while keeping thin static structures that delay schedules lose, as 3DGS-HPC also observes. Any one failing check would tell us which part of the claim to abandon.