Render-Back Correction for Feed-Forward Dynamic Gaussian Reconstruction

Fast 3D reconstruction systems for moving scenes, such as those used in driving, work as a one-way assembly line. A network first estimates depth and motion from camera images. A second stage turns those estimates into a set of 3D Gaussians: small colored ellipsoids that a graphics rasterizer can draw very quickly. Nothing ever checks the result against the images again. If the depth estimate is wrong behind a truck, the motion estimate built on that depth is also wrong, and the error passes silently into the 3D model. This note proposes closing that loop. Drawing Gaussians costs milliseconds, so the system can afford to render its own 3D hypothesis into a camera view it did not use for prediction, such as an overlapping neighboring camera or the next frame when it arrives. It then measures where the rendering disagrees with the real pixels and feeds that disagreement to a small learned network that corrects depth, motion, and the static/dynamic split.

In two sentences, the claim is this. In feed-forward dynamic reconstruction, errors from 2D estimates spread because the 3D stage never sees the raw pixels again. A few cheap rounds of "render the hypothesis, compare against a held-out view, apply a learned correction" can stop that spread within the latency budget of an online system. What is new is not rendering or iterative refinement on their own. It is using the fast rasterizer as a verification instrument inside a single forward pass, for scenes where things move.

Where the gap is

Errors that pass from 2D estimates into 3D outputs are documented across many tasks. In 3D human pose, models that lift 2D joints to 3D cannot recover joints the 2D detector missed. View Invariant 3D Human Pose Estimation normalizes poses to a canonical orientation, but its accuracy stays bounded by its input joints. KITRO shows that 2D noise spreads down the body's kinematic tree into impossible bone lengths. In 3D object localization, Byung-soo Kim et al. show that 3D recall is capped by the recall of the 2D detector. These papers motivate the proposal: they establish that a stage which discards the pixels cannot repair them.

The closest system to this proposal is DrivingScene. It reconstructs dynamic surround-view driving scenes feed-forward from two consecutive frames. It splits the scene into a static background and a "residual flow" for dynamic objects, and reports 28.76 dB PSNR on nuScenes at 0.21 s per surround-view suite. It is open-loop. If its depth network misjudges geometry, the flow prediction downstream fails, and nothing detects it.

The alternatives each give up something. Split4D avoids errors from 2D trackers by optimizing velocity-carrying Gaussians per scene, reaching 0.893 mIoU on Multi-Human. That per-scene optimization rules out online use. RobustSplat and RobustNeRF handle moving objects by discarding them. PCF-Lift and Trace3D down-weight noisy 2D masks, through feature covariance and through reverse rasterization respectively, but both assume a static scene and per-scene optimization.

The method this proposal borrows most directly comes from 3D-Aware Hypothesis & Verification (also; 3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation). That work proposes rigid rotation hypotheses for a single object's latent volume and scores each against the query, cutting CO3D angular error from 38.5° to 28.5°. This proposal replaces latent-volume scoring with actual rasterization into pixels. It also extends hypothesis-and-verification from one rigid object to whole scenes with non-rigid, multi-object motion.

Core Claim

The proposal makes three claims. First, a feed-forward dynamic Gaussian predictor equipped with a learned render-compare-correct loop will reduce error on dynamic regions and depth error on nuScenes compared with the same predictor without the loop. It should do so at a matched wall-clock budget and without per-scene optimization. Second, the gain comes specifically from the rendered disagreement signal, not from the extra compute or parameters the loop adds. Third, the loop should reduce how strongly upstream errors spread downstream: injecting a given amount of depth error should cause less degradation in motion and novel-view quality.

The third claim should be measured directly. If a deliberate depth perturbation of size $\epsilon$ produces downstream degradation $D(\epsilon)$, then the slope of $D$ near zero measures how strongly errors spread. The loop should flatten that slope.

Research Program

The base predictor is a DrivingScene-style network. It is retrained in our codebase and outputs, for each Gaussian, a position, covariance, color, static/dynamic probability, and 3D velocity for the dynamic part.

Verification views. Verification uses views the predictor did not use when forming each Gaussian. During training on surround rigs, each Gaussian is predicted mainly from its own camera, so it can be verified against the overlapping region of the adjacent cameras. Each Gaussian can also be verified at time $t{+}1$, by moving it along its predicted velocity and rendering into the next frame. At deployment, the next-frame check happens online: the state predicted at $t$ is forecast and rendered into frame $t{+}1$ as soon as that frame arrives, and the correction runs before the state is committed.

What the correction network sees. The disagreement is computed in two spaces. One is photometric: rendered color minus observed color. The other is semantic. Following Splat and Distill, which lifts DINOv2 features feed-forward into Gaussians and splats them to new views, each Gaussian carries a compact feature that can be rendered and compared against foundation-model features of the real image. The color residual responds to small geometric misalignments. The feature residual is less fooled by lighting changes and moving shadows, which the file notes as a confound for transient filters such as RobustSplat.

How residuals reach the Gaussians. Residuals are gathered back onto Gaussians through the rasterizer's per-pixel blending weights. This is the same reverse-rasterization idea Trace3D uses for mask ownership, applied here to residuals. A small attention-based update network then reads each Gaussian's gathered residual, its static/dynamic probability, and its neighbors. It outputs corrections to depth along the camera ray, to velocity, and to the static/dynamic probability. It runs for $K$ rounds, with $K$ between 1 and 3.

Why the split between static and dynamic matters. Consider a dynamic Gaussian whose next-frame residual is large while its cross-camera residual at the same time is small. That pattern points to a motion error, not a depth error, and the update network can learn this attribution. This is how the loop intercepts the depth-to-flow failure that DrivingScene's design leaves open.

Training. The loop is trained end-to-end. A loss on the final round's held-out-view rendering is combined with a smaller loss on every round, so that each round is useful by itself. Truncating at any $K$ then gives a valid latency/quality trade-off.

Role of the other directions. Geometric robustness enters as an evaluation lens. Novel views are rendered at lateral offsets from the real camera path, a stress test of whether geometry, not just appearance, is correct. Latency is a hard constraint: every comparison is reported at matched end-to-end milliseconds on the same GPU, with the cost of the render-back step itself reported.

Baselines

Do nothing. The naive reference is the base feed-forward predictor with no loop. It is retrained in our pipeline with identical data, resolution (352×640, six cameras), and training schedule. This isolates the loop's contribution from any differences in reimplementation.

Strongest published method. The strongest published competitor is DrivingScene. It is the only method in the file that does online, feed-forward, dynamic, surround-view reconstruction. It reports 28.76 PSNR, 0.895 SSIM, and 0.227 depth Abs Rel on nuScenes at 0.21 s, against 26.10 PSNR for Driv3R. We will report both its published numbers and our retrained number. Beating it means exceeding our retrained DrivingScene on overall PSNR and Abs Rel at no more than about 1.2× its latency. We also need a clearly larger margin on dynamic-region PSNR, since that is where the cascade hypothesis predicts the gain.

Test-time photometric optimization. The most dangerous competitor is DrivingScene followed by a few steps of ordinary gradient descent on the same held-out-view photometric loss. The number of steps is chosen to match our method's added wall-clock time. This is the obvious closed-loop alternative that needs no learned update network.

Confidence head without render-back. In this variant, inspired by PCF-Lift's covariance-based down-weighting, the base predictor additionally outputs a per-Gaussian uncertainty used to soften its own outputs. It receives a matched parameter budget, but no rendering ever enters it.

Key ablation: no residual input. The decisive ablation keeps the update network and its $K$ rounds unchanged, but replaces the gathered residuals with zeros. Everything else stays identical: parameters, compute, and access to the held-out frames as ordinary inputs. If this variant matches the full method, the render-back idea is not what helps.

Further ablations. We also ablate photometric-only versus feature-only residuals, and cross-camera-only versus next-frame-only verification.

Tuning budget. Every baseline receives the same hyperparameter sweep budget.

Why not just…?

Why not just run a few steps of test-time gradient refinement? This is a real threat. Gradient steps on a photometric loss are already a closed loop. Two arguments favor the learned update, but neither is decisive in advance. First, a single gradient step spreads blame through whatever blending weights currently exist, so it cannot distinguish a depth error from a motion error. The learned update sees both kinds of residual separately and can learn that attribution. Second, on driving data the photometric loss is noisy around moving shadows and specular surfaces. The wall-clock-matched comparison above settles the question. If gradient refinement equals our method at equal latency, the paper's claim reduces to "close the loop cheaply," and we would report it that way.

Why not just predict per-Gaussian uncertainty and trust low-confidence regions less? PCF-Lift shows that covariance-based down-weighting helps against noisy 2D segmentation. However, a confidence head predicted from the same inputs shares the blind spot of the depth network it is meant to judge. When depth is confidently wrong behind an occluder, nothing in those inputs reveals it. The pixels of a different view do. The confidence-head baseline tests this directly. Its expected failure mode is good calibration on blur and boundaries but no gain on occlusion-induced depth errors. We check this by splitting the evaluation by occlusion.

Why not just use a longer temporal window? DrivingScene's two-frame window is a stated limitation. More frames give more evidence, but they are still consumed open-loop. A longer window also increases latency and memory, both of which the file flags as bottlenecks for dynamic Gaussian methods. We will train a four-frame variant of the base predictor. If it matches the loop at equal latency, temporal context, not verification, is what matters. If the two combine additively, they are complementary.

Why not just per-scene 4D optimization such as 4D-GS or Split4D? These methods reach high quality: 34.05 dB on D-NeRF for 4D-GS, and 0.893 mIoU for Split4D. But they require minutes to hours per scene, and Split4D's own limitations note slow multi-stage training. They will be reported only as non-online upper references, not as competitors.

Proposed Experiments

Main evaluation. The main evaluation is on the nuScenes validation split (150 scenes), using DrivingScene's protocol. Metrics are PSNR, SSIM, and depth Abs Rel. We add PSNR restricted to dynamic regions, computed with masks derived from the 3D box annotations, and end-to-end latency per surround suite. The headline comparison is against the retrained DrivingScene and the test-time optimization baseline at matched latency, plotted as quality against $K$ and against milliseconds.

Cascade-sensitivity test. This is the most diagnostic experiment. We corrupt the base predictor's depth output with structured errors of increasing magnitude: smooth scale bias, random holes, and occluder-shaped dropouts. We then measure downstream velocity error and dynamic-region PSNR for the open-loop and closed-loop systems. The proposal predicts a visibly flatter slope with the loop, and a flatter slope than the no-residual ablation achieves. If the slopes match, the core claim fails regardless of headline PSNR.

Viewpoint robustness. We render at lateral offsets of 0.5 to 2 meters from the real camera path. No ground truth exists there, so we score cross-camera consistency in overlap regions and the agreement of rendered depth with lidar. These are reported as relative comparisons only.

Secondary instance-level check. On PandaSet, we attach instance features and evaluate whether the loop reduces the mask drift that the file notes for InstDrive. InstDrive is a per-scene method that reports 0.7535 mIoU single-camera and 0.7091 with six cameras; it serves as a reference rather than a matched competitor. No feed-forward novel-view-synthesis result on PandaSet was found on Lacuna.

Overhead accounting. We report the cost of the render-back step itself. The pass should stay in the millisecond range given hardware rasterization, but this must be measured, not assumed.

Decisive outcomes. The proposal succeeds if three things hold together: the full method beats DrivingScene on dynamic-region PSNR at matched latency, it beats the no-residual ablation, and it flattens the cascade-sensitivity curve. It fails, and should be reframed, if wall-clock-matched gradient refinement does equally well.