When people walk, drive, or wave their hands through a scene being captured on video, some of the background behind them may never be seen. Current methods for reconstructing such scenes do one of two things. Some build an explicit 3D model of the moving parts. Others throw the moving parts away as noise. Neither can say what lies behind a pedestrian who blocked the same patch of wall in every frame, so the reconstruction ends up with holes, torn surfaces, or floating specks. Image generators can paint in plausible content. Applied frame by frame, though, the painted content disagrees from view to view, and that disagreement turns into 3D artifacts.
We propose to use the reconstructed motion of the foreground itself to decide exactly where generation is allowed. The scene is represented with 3D Gaussian splatting, a method that models a scene as millions of small, semi-transparent colored ellipsoids that can be drawn very quickly. It is split into a static layer and a moving layer. As training runs, we keep an occlusion ledger: for every region of the static layer, we record how much it was actually seen, after accounting for transmittance and for occlusion by the moving layer across every frame. Regions the ledger marks as never observed are the only places where a depth-conditioned video diffusion model may supervise the reconstruction. A diffusion model is a generator that produces images by gradually removing noise. The same regions follow their own schedule for adding new ellipsoids. Everything hallucinated is static, so the finished model renders at static-layer speed, with no extra per-frame motion computation.
Core Claim
The claim is that the moving layer of a decomposed 4D reconstruction is the right source for the "where to hallucinate" signal. It beats 2D segmentation masks, outlier masks, and generation applied everywhere. Gating both diffusion supervision and ellipsoid growth on this ledger should resolve a tension the literature leaves open. RobustSplat shows that large transient errors trigger runaway cloning of ellipsoids, which later hang in mid-air as floaters, and that growth must therefore be delayed. Generative pipelines such as G4Splat and SGD deliberately inject high-variance gradients into unobserved regions, which is exactly what triggers that runaway growth. Our concrete prediction is that, on regions permanently hidden behind dynamic content, the gated method will do two things at once. It will beat distractor-removal methods on held-out views of those regions. It will also leave the observed static background and the moving foreground no worse than a decomposition baseline without generation.
Positioning
Three groups of prior work set up the gap. The first motivates the proposal. Distractor-free methods (3DGS-HPC, which masks out 16×16 patches flagged as distractors; DGGS, which prunes distractor Gaussians in a single feedforward pass; DeSplat, which absorbs distractors into per-view billboard layers) all state explicitly that background hidden in every view cannot be recovered. The first two name generative inpainting as the missing piece. These methods discard motion. We keep it and reuse it.
The second group supplies the representation. DeGauss separates a deforming foreground from a static background using a soft mask and renders at 157 FPS. SDD-4DGS learns a per-Gaussian static/dynamic coefficient. Split4D uses velocities to separate objects. None of these completes unseen background, and Split4D notes that removing objects leaves voids. We build on a DeGauss-style split because it already keeps the static layer free of deformation queries.
The third group supplies and constrains the generative component. G4Splat shows that ungated video diffusion produces multi-view inconsistency and scale drift. It fixes this with a visibility voxel grid and plane anchors, but only for static scenes, with visibility computed from camera coverage. SGD conditions diffusion on depth to regularize sparse driving trajectories and treats the world as static. AHOY! hallucinates occluded human body parts with a fine-tuned video diffusion model, but depends on a body template. Prior-Enhanced Gaussian Splatting explicitly cannot hallucinate unseen regions. We borrow G4Splat's idea of a visibility grid. Our visibility comes from integrating occlusion by the reconstructed moving layer over time, which no static method can compute. We also deliberately do not hallucinate the unseen sides of moving objects; that remains a separate, harder problem.
Finally, Monocular Dynamic View Synthesis: A Reality Check constrains the evaluation. Many dynamic benchmarks quietly leak multi-view information, so every result we report includes a setting free of camera "teleportation."
Research Program
We first train the decomposed model with delayed growth in the style of RobustSplat. This continues until the static layer stabilizes.
We then build the ledger. It is a sparse voxel grid over the static layer's bounds. For each training frame we render the static layer's transmittance, multiply it by one minus the moving layer's accumulated opacity, and add the result into each voxel along every ray. A voxel's score is its total unoccluded, transmittance-weighted exposure. Voxels below a threshold that lie inside the volume all cameras look at form the "never-seen" set.
In the third phase we render pseudo-views from perturbed cameras, as in SGD. We render the static layer only, together with its depth. A video diffusion model conditioned on that depth inpaints only the pixels whose rays pass through never-seen voxels. Pixels outside that set receive no generative gradient. Inside the never-seen set, growth of new ellipsoids is driven only by diffusion gradients and runs on a slower, capped schedule. Outside it, growth is photometric as usual. Hallucinated Gaussians carry a flag that lets us measure them separately and turn them off for diagnosis.
The ledger is refreshed a few times during training, because the moving layer improves as training goes on. Diffusion runs only during training. At render time the model is the same two-layer splat as DeGauss with more static ellipsoids, so we expect its frame rate to stay close to that baseline. We will measure this rather than assume it.
Baselines
The naive reference point is the decomposition baseline itself, DeGauss trained on the same frames with no generation. It leaves never-seen regions empty or smeared. Any gain we report must be measured against it in two places: the never-seen region, and the observed region, which must not degrade.
The strongest distractor-removal competitors are RobustSplat and 3DGS-HPC. They are the best static-recovery methods in the file and represent what "just discard the movers" achieves. We run their released settings on identical frames with the same iteration budget. For reference, 3DGS-HPC reports PSNR gains of 1.39 to 5.54 dB over vanilla 3DGS on RobustNeRF with roughly 13–17 minutes of training on an RTX 4090. RobustNeRF itself reports 30.87 dB on its distractor scene. For PSNR restricted to permanently occluded regions, no reported result was found on Lacuna for any method.
The strongest generative competitor is a two-stage pipeline: train DeGauss, then run G4Splat-style completion on the static layer, with visibility computed from 2D distractor masks. This is the most serious threat, because it uses the same ingredients without coupling them. For reference, G4Splat reports 23.90 dB PSNR and 6.61 Chamfer distance on 5-view Replica, but those numbers are for static scenes.
To isolate the key ingredient, we run three ablations. The first replaces the ledger with 2D-mask-derived visibility. The second removes gating so diffusion supervises all pseudo-view pixels. The third removes the separate growth schedule.
All methods share frames, camera poses, the diffusion model and its denoising budget where one is used, GPU type, and tuning trials per method. Beating the competitors means four things. On never-seen regions we must gain in both PSNR and LPIPS. On observed regions we must stay within noise. We must show fewer floater ellipsoids, counted in empty space. Rendering must stay within a small margin of DeGauss's 157 FPS on an RTX 4090.
Why not just…?
Why not just inpaint the input video in 2D and then run static 3DGS? This is cheap, and with a strong video diffusion model it might yield clean frames. G4Splat's results show that per-frame generative content lacks 3D consistency and drifts in scale, which produces floaters once it is lifted to 3D. The approach also deletes the moving foreground, which is often the point of capturing it. We include this pipeline, using the same diffusion model for inpainting, as a baseline. If it matches us on never-seen regions and on cross-view consistency, the ledger is not worth its complexity, and we will say so.
Why not just run DeGauss and then complete the holes afterward? This is the real threat. Our method differs in two ways. Visibility comes from the moving layer rather than from 2D masks, and growth is gated during training rather than added afterward. The mask-visibility ablation and the two-stage baseline separate the two contributions. If the two-stage pipeline using our ledger matches the joint version, the honest conclusion is that the ledger is the contribution and joint training is not. That would still be a useful result.
Why not just mask movers with SAM2 and let diffusion fill the masked pixels? 3DGS-HPC reports that semantic masks miss shadows and other non-object transients and can delete textureless walls. Split4D reports that video trackers drift under fast motion. Moreover, a pixel that is masked in one frame may be visible in another, and 2D masks cannot tell which. The integrated ledger can. The mask-visibility ablation tests this directly, so it settles the question.
Proposed Experiments
We need ground truth for regions hidden in every training view, so we construct it.
On Neural 3D Video, a multi-camera dynamic dataset, we choose camera subsets in which the actor permanently hides a background patch that a held-out camera sees. We then evaluate on that held-out camera, splitting the error by never-seen and observed masks. On this dataset SDD-4DGS reports 32.39 dB at about 85 FPS, and 4DGS-1K reports over 1000 FPS. These serve as context for full-frame quality and speed.
On RobustNeRF and NeRF On-the-go, we choose training subsets in which a distractor covers the same background in every selected view. The clean test views supply ground truth. DGGS's 21.02 dB on Statue and NeRF On-the-go's 21.41 dB on Patio are full-image references.
On the DyCheck iPhone benchmark, we evaluate monocularly without teleporting cameras, following the Reality Check protocol. Prior-Enhanced Gaussian Splatting's 17.63 dB is the reference there, and this setting tests whether the ledger survives a noisy moving layer.
The decisive checks are four. Gating must improve never-seen-region LPIPS over the ungated ablation while leaving the floater count no higher than DeGauss. Ledger visibility must beat mask visibility on the never-seen region. Observed-region PSNR must not drop relative to DeGauss. Frame rate must stay near the static-layer baseline. If gating fails the floater test, the RobustSplat tension is not resolved and the method should not be claimed.
References
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- G4Splat: Geometry-Guided Gaussian Splatting with Generative Prior
- SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- Distractor-free Generalizable 3D Gaussian Splatting
- DeSplat: Decomposed Gaussian Splatting for Distractor-Free Rendering
- DeGauss: Dynamic-Static Decomposition with Gaussian Splatting for Distractor-free 3D Reconstruction
- SDD-4DGS: Static-Dynamic Aware Decoupling in Gaussian Splatting for 4D Scene Reconstruction
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- AHOY! Animatable Humans under Occlusion from YouTube Videos with Gaussian Splatting and Video Diffusion Priors
- Prior-Enhanced Gaussian Splatting for Dynamic Scene Reconstruction from Casual Video
- Monocular Dynamic View Synthesis: A Reality Check