Correcting Depth, Not Texture: Ray-Constrained Geometric Priors for Monocular 4D Reconstruction

When a single handheld camera films a moving person or animal, every pixel pins down a line of sight but not how far along that line the surface sits. Current methods that rebuild the moving scene in 3D fit the images well and still get the shape wrong: limbs detach, bodies look squashed front-to-back, surfaces shear. Methods that bring in image generators to fix these reconstructions mostly repaint what the viewer sees. They sharpen textures in poorly seen regions and leave the broken geometry underneath.

We propose a different use of the generator. A diffusion model (a generator that learns to turn random noise into realistic images by removing noise step by step) looks at rendered surface-normal maps of the moving object from new viewpoints and says which way the shape should move. We then allow each well-observed point to move only along its own line of sight, the one direction the input video cannot constrain. Points that were never observed may move freely. The prior can therefore repair depth structure without undoing what the video already established, and it can complete hidden surfaces without spraying stray geometry into the scene.

Core Claim

In monocular dynamic reconstruction, the error that generative priors should target is the depth null-space: the set of 3D configurations that render identically into the input frame. We claim that projecting a geometry-only diffusion gradient onto this null-space, per point and per time step, will reduce depth and shape error substantially relative to both RGB-based diffusion refinement and depth-prior-only methods, with no loss of fidelity on observed pixels.

Our reading of the literature supports the gap. ViDAR and UA-4DGS both apply personalized image diffusion to monocular 4D Gaussians. ViDAR supervises dynamic regions with pseudo-multi-view renders. UA-4DGS restricts diffusion supervision to regions its 2D uncertainty map flags as uncertain. Both act through RGB, and ViDAR's stated limitation is that it cannot fix detached or sheared geometry in its scaffold. MoDGS attacks depth directly with an ordinal loss that survives the scale drift of monocular depth models, but it has no way to complete unseen surfaces. NeuSD shows that distilling diffusion priors through rendered normal maps rather than RGB separates shape from appearance and yields cleaner completions, but only for static objects represented as implicit surfaces. Our proposal takes the normal-map supervision from NeuSD and the ordinal constraint from MoDGS, and adds the one new ingredient: gradient routing that confines the prior to the ambiguous direction for observed geometry.

Mechanism

We build on a dynamic 3D Gaussian Splatting backbone. This represents the scene as many small colored 3D blobs rendered by fast rasterization. The blobs live in a canonical frame and are moved to each timestamp by a learned deformation, as in Deformable 3D Gaussians, or by a point-track motion scaffold of the kind used by ViDAR and ProDyG.

At each training step we sample a timestamp $t$ and a novel viewpoint. We render the dynamic region's normal map from that viewpoint and pass it to a frozen pretrained image diffusion model through normal-map score distillation, following NeuSD, which includes its frozen-noise stabilization. This produces a gradient $g_i$ on the time-$t$ position of Gaussian $i$.

Let $r_i(t)$ be the unit vector from the input camera at time $t$ through that Gaussian. Let $v_i(t)\in[0,1]$ be its visibility in the input frame, computed from its accumulated alpha-blending weight. We apply

$$ \tilde g_i = v_i(t)\, r_i(t)\, r_i(t)^\top g_i + \big(1 - v_i(t)\big)\, g_i , $$

so that visible Gaussians receive only the along-ray component and occluded Gaussians receive the full gradient. The projected gradient flows back into the deformation field and canonical positions.

Two further rules keep the prior honest. First, adaptive densification (the step that adds new Gaussians where error is high) is disabled for gradients coming from the generative loss on visible Gaussians. This follows the finding of RobustSplat that inconsistent error signals trigger densification and leave permanent floaters. Second, MoDGS's ordinal depth loss stays active on observed pixels. The monocular depth model then contributes only depth order, which is reliable, while the diffusion prior contributes shape, and neither is asked to supply absolute scale.

Research Program

The work has three parts. The first builds the system on a single backbone and fixes the compute budget to roughly match ViDAR's reported cost of about 4 GPU hours per sequence. This keeps comparisons honest about cost.

The second part is a controlled study of whether null-space routing is what matters. We compare four variants that differ only in how the generative gradient is used:

  • unprojected normal-map distillation;
  • projected normal-map distillation;
  • projected RGB distillation;
  • no distillation.

The third part measures geometry directly rather than through image scores alone. The Reality Check study showed that masked image metrics on DyCheck can reward reconstructions that look right from nearby views while being structurally wrong.

Baselines

The naive reference point is the backbone alone, trained with photometric loss plus the MoDGS ordinal depth loss and no generative prior. It shows how much structural error remains after depth priors have done what they can. On DyCheck we also report a plain 4D Gaussian model, for which UA-4DGS reports 14.14 dB masked PSNR.

The strongest existing competitors are UA-4DGS and ViDAR, re-run on the same backbone where possible and with the same per-sequence GPU budget:

  • UA-4DGS reports 15.25 dB masked PSNR on DyCheck, the best absolute DyCheck number in our file.
  • ViDAR reports only a roughly 1 dB PSNR gain over the previous best on DyCheck; no absolute ViDAR DyCheck figure was found on Lacuna.

These are the right competitors because they are the closest prior work: generative diffusion applied inside monocular dynamic Gaussians. They are also exactly the methods whose geometry we claim stays broken. Beating them means two things together. Depth error against DyCheck's LiDAR depth on the dynamic region must drop clearly, and masked PSNR and LPIPS must be no worse than theirs. An image-metric gain without a depth gain would not support the claim.

For geometric completion of a non-rigid animal, AnimatableDreamer reports 3.65 cm Chamfer distance on Cat-Coco against 10.7 cm for BANMo. It uses a learned skeleton and multi-view diffusion, so matching it without a skeleton would be meaningful. CogNVS reports 22.63 dB PSNR on Kubric-4D, and it serves as the feed-forward alternative.

The key ablation replaces the projection with the unprojected gradient. A second ablation swaps normal maps for RGB while keeping the projection. Together these isolate the two ingredients, the geometric signal and the null-space routing.

Why not just…?

Why not just use a stronger or more temporally consistent monocular depth prior? This is the most serious threat. If depth order from a video-consistent depth model already fixes squashed or sheared bodies, the diffusion prior only matters for hidden surfaces. MoDGS shows that absolute depth supervision fails because of frame-to-frame scale drift, and that ordinal loss helps. But ordinal constraints are silent about the backsides of subjects and about shapes whose depth order is right while proportions are wrong. Our depth-prior-only baseline tests this directly. If its LiDAR depth error on visible dynamic pixels matches our method's, we would narrow the claim to completion of unseen geometry.

Why not just apply RGB diffusion refinement as ViDAR does? ViDAR itself states that its refinement sharpens texture over failed geometry. The projected-RGB ablation checks whether that failure comes from the RGB signal or merely from the lack of routing. If projected RGB distillation fixes depth as well as projected normals, then routing is the whole contribution. In that case the method should be simplified to that.

Why not just add unconstrained normal-map distillation to dynamic Gaussians? This is the obvious port of NeuSD to dynamic scenes. We expect it to trade fidelity for completion: moving visible points sideways breaks the photometric fit, and generative error triggers floaters, as RobustSplat documents. This is exactly the unprojected ablation. If it performs as well as the projected version on both depth error and masked PSNR, our core claim fails.

Why not just inpaint novel views with a video diffusion model and skip the 3D optimization? CogNVS does this well within about 90 degrees of the capture path. However, it depends on an accurate 2.5D reprojection and does not produce a persistent 4D asset, so depth errors in the lifting step propagate into its holes. We compare against it on Kubric-4D and DyCheck at wide viewpoint deviations, where we expect the gap to be largest.

Proposed Experiments and Decisive Checks

The decisive experiment is synthetic and controlled. On Kubric-4D, where ground-truth geometry exists, we initialize dynamic objects with depth deliberately scaled or sheared along camera rays by known amounts. This corrupts shape while leaving the input-view renders unchanged. We then measure how much of the injected error each variant removes, using per-object Chamfer distance and depth error, while tracking input-view PSNR.

The claim predicts three results:

  • The projected normal variant recovers most of the injected error and holds input-view PSNR steady.
  • The unprojected variant recovers some error but loses input-view PSNR.
  • The RGB variants and the depth-only baseline recover little.

On real data, we evaluate on DyCheck under co-visibility masked PSNR, SSIM, and LPIPS. We add PCK-T for correspondence accuracy and LiDAR depth error restricted to dynamic pixels, since the file lists metric LiDAR depth as part of that dataset. We also report Chamfer distance on Cat-Coco for full-shape completion. We break DyCheck results down by the Effective Multi-view Factors from the Reality Check study. The benefit should be largest on sequences with little camera motion, where the null-space is widest.

A failure mode we will report openly is the Janus problem: when camera motion is minimal, the generator hallucinates front features on the back. We will count its incidence on low-motion sequences.