Attributing Render Errors in Feed-Forward 4D Reconstruction

Fast 3D reconstruction networks now turn a short video into a renderable scene in under a second. A simple trick makes them sharper: render the current guess, compare it with the input frames, and feed the error image back into the network for a few more update steps. In a moving scene, though, that error image is ambiguous. A pixel can be wrong for three different reasons. The camera position may be slightly off. An object may have moved in a way the model has not captured. Or the pixel shows a surface that no input frame ever saw. If all three kinds of error drive the same update, the network nudges the camera to explain a person's moving arm, or grows stray blobs in mid-air to explain a hole behind them.

We propose a recurrent refiner that first decides which of the three causes produced each pixel's error and then routes that error to a separate update. Camera-explained error refines the camera pose, coarse resolution first and fine resolution later. Motion-explained error refines the scene's motion. Error from never-seen surfaces updates nothing in the geometry and is exported as a calibrated "never observed" mask that a generative model can fill. Our claim is that this attribution step, more than added model capacity, is what lets learned error feedback carry over from static scenes to casual dynamic video. It should give more accurate poses, fewer floating artifacts, and a disocclusion mask that is better than the opacity heuristics used today.

Core Claim and Positioning

The method we borrow comes from ReSplat: Learning Recurrent Gaussian Splats. ReSplat represents a scene as 3D Gaussians, small colored blobs that are splatted onto the image. It refines their attributes over four steps, each driven by rendered photometric and feature error maps. On DL3DV 8-view synthesis it beats the single-pass DepthSplat by 3.5 dB PSNR. ReSplat assumes a static scene and ground-truth poses, so every residual has only one possible cause. The research file names exactly this ambiguity as an open tension: in dynamic scenes, error feedback "could arise from incorrect camera pose, unmodeled dynamic motion, or disoccluded unobserved background." No existing feed-forward model resolves it.

Several neighboring systems handle pieces of the problem. Two feed-forward 4D models predict camera poses and motion in one pass but never refine against their own rendering error. Any4D predicts depth, poses and global scene flow. 4RC predicts base geometry plus displacements. Both are our natural initializers, not our competitors in kind.

Two per-scene optimizers show that pose refinement must ignore moving pixels, but both cost tens of minutes per video. 4D3R does this with masked PnP-RANSAC and motion-aware bundle adjustment, at about 50 minutes on HyperNeRF. RoDyGS optimizes static Gaussians jointly with the camera trajectory. These papers motivate the static/dynamic split.

Static pose-free models motivate the coarse-to-fine schedule. PF3plat does coarse-then-fine pose alignment feed-forward but only for static scenes. The broader pose literature explains why coarse levels widen the basin of convergence and fine levels add precision: neural feature fields for pose refinement and LATITUDE's low-pass filtering both establish this.

RobustSplat supplies a cautionary mechanism. It shows that unexplained error spikes trigger Gaussian growth that permanently scars the background. Our "never observed" channel is a learned, feed-forward version of its idea of keeping unexplained error out of geometry growth.

On the generative side, three systems fill disocclusions. FreeGen uses rendered opacity as the confidence mask for video diffusion, but assumes known odometry poses. FreeOrbit4D chains SAM2, VGGT, SV4D and a video diffusion model. Gen3R does so only for static scenes. We do not try to beat these as generators. Our contribution to them is a better answer to "where should the generator paint?" in dynamic, pose-free video.

Research Program

The model starts from a feed-forward 4D initializer, Any4D or 4RC, that outputs per-frame depth, camera poses, and per-point 3D motion. These are converted into a static Gaussian set and a dynamic Gaussian set, in the spirit of RoDyGS's split. At each recurrent step $t$, the current scene is rendered into every input frame, and photometric and feature residuals are computed at pyramid level $\ell(t)$. The schedule runs from coarse to fine.

A small attribution head reads the residuals, the rendered depth and opacity, and the current motion field. For each pixel $p$ it outputs soft weights $\pi_{\text{cam}}(p), \pi_{\text{mot}}(p), \pi_{\text{new}}(p)$ that sum to one. These are the probabilities that the error comes from camera pose, object motion, or a never-observed surface. Each update block then sees only its share of the error:

$$ \Delta\xi_t = g_{\text{cam}}\big(\pi_{\text{cam}} \odot r_t^{\ell(t)}\big),\quad \Delta m_t = g_{\text{mot}}\big(\pi_{\text{mot}} \odot r_t^{\ell(t)}\big), $$

Here $r_t^{\ell(t)}$ is the residual map at the scheduled resolution, $\xi$ is the set of camera poses, $m$ is the dynamic motion field, and the $g$ functions are learned update networks. The never-observed share $\pi_{\text{new}} \odot r_t$ drives no geometric update. It is accumulated across steps into an output mask. The coarse-to-fine schedule matters for the camera branch in particular. Early, low-resolution residuals tolerate large initial pose errors from the initializer. Late, high-resolution residuals give precision. Only pixels the head believes are static and visible contribute to the pose.

Supervising the attribution head is the crux of the project. Synthetic data provides exact labels. Kubric, which DGS-LRM already uses at 40,000 scenes, and PointOdyssey both give ground-truth poses, scene flow and depth. From these, every pixel of a rendered error can be labeled. It is camera-caused under injected pose noise. It is motion-caused where ground-truth flow is nonzero and the model's motion is wrong. It is never observed where ground-truth depth shows the surface is visible in no context frame. We will train on synthetic data with these labels plus a rendering loss. On real casual video we will continue with the rendering loss alone, and the attribution head is regularized to stay consistent with how its predictions behave under synthetic pose perturbations.

The output is a refined 4D Gaussian scene with poses and a disocclusion mask. For extrapolated views we pass this mask, in place of rendered opacity, to a frozen video diffusion completion stage of the FreeGen kind. That way any gain in generation can be credited to the mask.

Baselines

The naive reference point is the initializer alone, Any4D or 4RC, rendered without any refinement. It measures how much the recurrence adds at all. For dynamic novel view synthesis the strongest feed-forward competitors are the following two systems.

BTimer reports 25.82 dB PSNR on the NVIDIA Dynamic Scene dataset at 150 ms, using DROID-SLAM poses. We will run it with the same frames and resolution (12 context frames, 256×256). Beating it means exceeding 25.82 dB while running without a separate SLAM stage.

DGS-LRM reports approximately 14.89 PSNR on the DyCheck iPhone benchmark and an ATE-3D of 0.21 on PointOdyssey. Beating it means higher iPhone PSNR, the regime where Monocular Dynamic View Synthesis: A Reality Check shows methods lose 4–5 dB, together with lower ATE-3D.

For geometry and motion, Any4D reports APD 93.44 on Dynamic Replica. We need to at least match it, since we start from it, and the question is whether refinement raises it. On TUM-dynamics, 4RC reports the best ATE among compared methods. No reported numerical value was found on Lacuna for 4RC's ATE, so we will rerun it under our protocol.

The quality ceiling is per-scene optimization. NSFF reports 28.19 PSNR on NVIDIA after two days per scene. 4D3R reports up to 1.8 dB over SC-GS on HyperNeRF at about 50 minutes per scene. We will report the time-quality tradeoff against these rather than claim to beat them.

The key ablations share the backbone, the four recurrent steps, the training data and the compute budget. The first is the unattributed recurrence: ReSplat's scheme extended to 4D, with the full residual driving every update. The second is heuristic attribution. It replaces the learned head with SAM2-style motion masks for the dynamic share and rendered opacity below a threshold for the never-observed share, which is the FreeGen convention. The third is attribution without the coarse-to-fine schedule, with all steps at full resolution. For extrapolation quality, FreeGen's ±4 m nuScenes result (FID 14.44, FVD 70.75) is the reference. On our dynamic benchmarks we will compare the same diffusion stage driven by opacity masks against driven by our mask.

Why not just…?

Why not just use Any4D's feed-forward poses and stop? This is the most serious threat. Any4D already predicts poses in one pass, and its APD lead is large. The answer must be empirical. If attributed refinement does not measurably lower ATE on TUM-dynamics and iPhone over Any4D's raw poses, then the pose branch is unnecessary, and the project reduces to attribution for disocclusion masking alone. We report Any4D-only, Any4D plus unattributed refinement, and Any4D plus attributed refinement side by side.

Why not just extend ReSplat to video and let the network sort out the residuals implicitly? A big enough recurrent network might learn the separation without explicit heads. The unattributed ablation tests exactly this at matched parameters. The evidence that it may fail is indirect: RobustSplat and 4D3R both found that letting transient or dynamic error reach static updates corrupts geometry and pose. If the implicit version matches ours, we will say so. Explicit attribution would then be worth only its interpretable mask.

Why not just use SAM2 masks and low opacity as the attribution? Heuristic masks are cheap and FreeGen shows opacity works for driving. Two problems remain. Opacity is low wherever the model is uncertain, not specifically where surfaces were never seen. And motion segmentation does not catch pose-induced error on static surfaces. We will measure mask quality directly against ground-truth never-seen visibility on Kubric and PointOdyssey, and pose quality with each mask source. If heuristics come within noise, the learned head is not justified.

Why not just run per-scene optimization like 4D3R or RoDyGS? These methods are the quality ceiling. They take tens of minutes to days per scene, and by RoDyGS's own account they cannot recover surfaces that no frame observed. The proposal targets the sub-second regime and a mask that tells a generator where to work, so we compare on the time-quality curve rather than on raw PSNR alone.

Decisive Checks

The first check tests the basin of convergence. On Kubric and PointOdyssey, we perturb the initial poses by controlled rotations and translations. We measure the final pose error and the fraction of sequences that converge, for attributed versus unattributed refinement and with versus without the coarse-to-fine schedule. Attribution should widen the basin mainly when moving objects cover a large share of the frame.

The second check is a leakage test. We measure how much pose error comes back when ground-truth never-observed pixels are deliberately routed into the camera branch. This directly tests whether disoccluded pixels corrupt pose, which is the mechanism behind the core claim.

The third check is mask calibration. We report precision and recall of the never-observed mask against synthetic ground truth, compared with opacity thresholds. Downstream, we measure the FID/FVD change from substituting our mask into the same diffusion stage on held-out DyCheck views.

The project succeeds if attribution beats the unattributed and heuristic variants on pose error and iPhone PSNR at equal compute. It fails cleanly if Any4D's raw poses plus opacity masks match it.