A single moving camera can measure how far away the static parts of a scene are, because camera motion makes them shift against each other (parallax). It cannot do the same for a person or object that moves at the same time, because that object's own motion is confused with the camera's. Recent systems for synthesizing new viewpoints of dynamic scenes from phone video hide this weakness. They first build a rough 3D or 4D scaffold, which is a point cloud or a set of 3D Gaussians (small colored blobs rendered by splatting). They render that scaffold from the requested viewpoint and ask a video diffusion model, a generator that learns to turn noise into realistic frames, to clean up the result. The diffusion model treats every scaffold point as ground truth, so a limb placed 30 cm too far along the camera ray is rendered faithfully and in high detail. ViDAR names this failure directly: the diffusion stage "renders bad geometry with high visual detail" (ViDAR).
We propose to give the generator the ambiguity instead of hiding it. Each uncertain scaffold point is passed to the diffusion model as a depth interval along its original camera ray, not as a single point. Seen from a new viewpoint, that interval projects to a short streak. The generator is trained to pick the place on the streak where the object plausibly sits, and its choice is fed back to shrink the interval in the 4D model. The geometric fact that makes this work is simple. The direction that one camera cannot measure, along its own line of sight, becomes a sideways direction in any view rotated away from it, and sideways position is exactly what an image generator is good at judging. Segmentation masks are not needed either: points whose intervals collapse under ego-motion parallax are static, points whose intervals stay wide are dynamic, and this split takes the place of the foreground masks that current pipelines borrow from external trackers.
Core Claim
Our central claim is that monocular 4D novel-view synthesis improves, in both geometry and appearance, when the video generator is conditioned on explicit per-point depth intervals and its outputs are used to resolve those intervals. Only correcting scaffold depth along the ambiguous axis can fix structural errors such as detached or displaced limbs. Re-texturing renders cannot, and neither can treating uncertain regions as holes to inpaint. A secondary claim is that the same interval widths give a static/dynamic decomposition competitive with mask-based ones. This would test, in the monocular dynamic setting, the finding from 3DGS-HPC that coarse, consistency-derived masks can beat pixel-accurate semantic masks.
Research Program
The scaffold. We start from a standard monocular dynamic Gaussian scaffold built on MegaSAM-style camera poses and video depth, the same starting point used by CogNVS. Each Gaussian center is reparameterized as $\mu = o_s + d\, r_s$, where $o_s$ and $r_s$ are the origin and ray direction of the source frame $s$ that first observed it. Its depth $d$ is constrained to an interval $[d_{lo}, d_{hi}]$.
Interval widths. We initialize the widths from the closed-form photometric variance along the line of sight introduced by USplat4D. That work showed monocular 4D uncertainty is strongly anisotropic and largest along the optical axis. It then used the uncertainty only to regularize motion through neighborhood graphs. We use the same quantity as a signal passed to the generator. Static points seen across a baseline get narrow intervals. Points that move with or against the camera keep wide ones.
Rendering intervals for the generator. To render the scaffold into a target view, we splat each Gaussian at its interval midpoint as usual. We add an extra conditioning channel that draws the projected interval as a streak, encoding the source-ray direction and the interval length. The generator is a CogVideoX-based inpainter of the kind CogNVS adapts, with this extra channel injected into its condition branch. The design follows how Diffusion as Shader feeds 3D coordinate videos into a condition branch built from DiT (diffusion transformer) blocks. Diffusion as Shader assumes those coordinates are correct and reports hallucinations when tracking fails. Our channel tells the model where the coordinates might be wrong and in which direction.
Training the generator. We train it self-supervised on ordinary videos with camera motion. A scaffold is built from frame $s$, and selected points are deliberately displaced along their source rays by a sampled amount. The scaffold is rendered into frame $t$ with the true displacement range drawn as the interval channel. The model must reconstruct the real frame $t$. To succeed, it has to learn that the correct pixel lies somewhere on the streak and use image evidence to choose where. Synthetic multi-view data from Kubric-style scenes adds true wide-baseline supervision. Kubric-4D itself is held out for evaluation.
Closing the loop. At test time we render a ring of novel views, generate them, and refit the scaffold. Only the per-Gaussian ray depth $d$, plus appearance, may move, and $d$ is clipped to its current interval:
$$\min_{\{d_i\}} \sum_{v} w_v \,\big\| \mathcal{R}_v(\{o_{s_i} + d_i r_{s_i}\}) - \hat I_v \big\|_1 \quad \text{s.t. } d_i \in [d_{lo,i}, d_{hi,i}].$$
Here $\mathcal{R}_v$ is the differentiable splat render into view $v$, $\hat I_v$ is the generated frame, and $w_v$ grows with the angle between view $v$ and the source rays, since that angle sets how much the depth axis appears sideways in the image. After each round, intervals shrink around the refit depth according to how consistently the generated views agree. Iteration stops once the intervals stop shrinking.
Constraints from prior work. Two results limit the design. ViDAR and Vivid4D show that open-loop enhancement cannot fix structure, which motivates the loop. De4D-SLAM's "optimization paradox" warns that letting dynamic parameters absorb static residuals blurs static geometry (De4D-SLAM). For that reason we freeze narrow-interval (static) Gaussians during refitting, in the spirit of its stop-gradient on static residuals.
Baselines
Naive reference. The do-nothing reference is the initial scaffold rendered directly, with no generator. It is the MoSca-style or USplat4D-style dynamic Gaussian model. USplat4D reports a DyCheck PSNR of 19.63 at 2x resolution, against 19.32 for MoSca. This tells us how much any generative stage adds at all.
Strongest existing method. The strongest competitor is CogNVS (Reconstruct, Inpaint, Test-Time Finetune). It uses the same scaffold source and the same generator family, and it already inpaints missing regions with test-time finetuning. It reports 22.63 PSNR on Kubric-4D, against 20.93 for TrajectoryCrafter and 18.59 for GCD, with about 140 minutes of finetuning per 400-frame video. We will run CogNVS and our method from identical MegaSAM scaffolds. Both get the same CogVideoX backbone, the same pretraining video corpus, and the same test-time budget, with our loop rounds counted inside the 140 minutes. Beating it means a reliable Kubric-4D PSNR gain over CogNVS run under our protocol, plus a clear reduction in dynamic-region depth error against Kubric ground truth. CogNVS does not report the depth-error metric, and it is the one that tests our claim.
Other generative baselines. ViDAR is the closest diffusion-enhanced 4D baseline on DyCheck. It reports roughly 1 dB over the previous best but no scalar we can quote; no reported result found on Lacuna for an exact ViDAR DyCheck PSNR. We will rerun it with its DreamBooth stage at its published budget. Vivid4D reports DyCheck mPSNR 12.34 and mLPIPS 0.4516, and serves as the inpainting-based reference. CAT4D represents the scaffold-free approach and reports 21.97 PSNR on NVIDIA Dynamic Scenes for fixed viewpoints with varying time (CAT4D).
Ablations. The ablations isolate the key ingredient. The first, hard-hole, keeps our loop but replaces the interval channel with a binary mask that deletes high-uncertainty points and asks the model to inpaint. The second, no-interval loop, refits all Gaussian parameters freely from generated views without the ray constraint. The third, open-loop interval, uses the interval channel but runs no refitting.
Decomposition baselines. For static/dynamic decomposition we compare interval-thresholding against SAM-mask splitting of the same scaffold. Split4D's 0.893 mIoU on Multi-Human comes from multi-view input, so we cite it as an upper reference and not as a direct competitor (Split4D).
Why not just…?
Why not just mark uncertain pixels as holes and let an inpainter like CogNVS fill them? This is the most serious threat, and the hard-hole ablation exists to settle it. Deleting an uncertain limb throws away the information that the limb exists and lies somewhere along a known ray. The inpainter must then reinvent its shape and position from appearance alone. That is exactly the setting where CogNVS reports failures under extreme viewpoints and scaffold degradation. Our prediction is that hard-hole matches us on PSNR for small view changes but loses on dynamic-region depth error and on large rotations. If hard-hole matches us on Kubric-4D depth error too, the interval channel is unnecessary, and we would report the work as a closed-loop refinement study instead.
Why not just iterate render, enhance, and refit without any uncertainty, as a closed-loop ViDAR? Without the ray constraint, refitting can change anything, including static geometry, which invites De4D-SLAM's optimization paradox. The generator was also conditioned on the wrong geometry, so it tends to confirm that geometry. The no-interval loop ablation measures both effects directly. We expect static-region PSNR to degrade and the dynamic depth error to stall near its initial value.
Why not just use a better video depth model or SAM masks plus USplat4D? Better per-frame depth does not remove the ambiguity. When an object moves during the capture, one camera simply has no parallax for it. USplat4D already uses the best geometric regularization available and still fails at 120 to 180 degree view shifts, where only a prior on appearance can choose depth. SAM-based splitting fails on shadows and subtle clutter, as 3DGS-HPC documents, and tracker-dependent splitting scores far below tracker-free splitting on Multi-Human (0.592 against 0.893 mIoU in Split4D). Our decomposition experiment compares SAM splitting with interval splitting on the same scaffold.
Proposed Experiments
Experiment 1: depth correction on synthetic data. The decisive experiment uses Kubric-4D, where ground-truth depth exists. We measure the median depth error of dynamic points along the source ray before and after each loop round, for our method and the three ablations. We also report novel-view PSNR, SSIM and LPIPS against CogNVS's 22.63. The claim holds only if the full method reduces dynamic depth error substantially more than the hard-hole and no-interval variants, and the reduction grows with the target-view rotation angle. The second condition is the signature of the sideways-axis mechanism. If gains do not grow with angle, the improvement comes from somewhere else.
Experiment 2: real-world viewpoint synthesis. On DyCheck iPhone, we report PSNR, SSIM and LPIPS overall and on the dynamic-region subset. The comparisons are against USplat4D (19.63), Vivid4D (12.34 mPSNR), and reruns of ViDAR and CogNVS. On NVIDIA Dynamic Scenes we compare with CAT4D. To probe the regime where monocular ambiguity dominates, we use the articulated Objaverse benchmark from USplat4D at 120 to 180 degree view shifts.
Experiment 3: mask-free decomposition. On Bonn and DyCheck, we compare interval-thresholded static/dynamic labels against SAM masks and against WildGS-SLAM's learned uncertainty (WildGS-SLAM). We evaluate both mask quality and the effect of each split on static-background PSNR.
Failure analysis. We report results stratified by motion type. Motion toward or away from the camera is the hardest case for monocular depth and should benefit most. Sideways motion should benefit least, because sideways displacement is already visible in the original images.
References
- ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputs
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos
- Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction
- Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control
- Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpainting
- De4D-SLAM: Gradient-Isolated Static-Dynamic Decoupling for Monocular SLAM in Dynamic Environments
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- WildGS-SLAM: Monocular Gaussian Splatting SLAM in Dynamic Environments