Space-Time Plane Sweeps for Feed-Forward Dynamic Reconstruction from Sparse Camera Rigs

Systems that turn a handful of photos into a 3D scene in one network pass, without fitting a model to each scene, depend on a simple geometric test. If two cameras see the same surface point, their image features should match at the correct depth. MVSplat turns that test directly into geometry. It places 3D Gaussians (small colored ellipsoids that can be drawn very quickly) at the depth where views agree, and it sets each Gaussian's opacity from how confident that agreement is. Free space never produces a confident match, so it gets no "floaters", the semi-transparent junk that plagues sparse-view reconstruction. The test assumes nothing moves. When a person walks or a car drives between frames, false matches corrupt depth. Today's dynamic feed-forward models (StreamSplat, DGS-LRM, 4RC) respond by dropping the matching test altogether. They lean on single-image depth networks or on attention across time, and they lose the floater protection.

We propose to keep the matching test and split it along time. With a small synchronized rig of two to four cameras, matching across cameras at the same instant is valid for everything in the scene, moving or not. Matching against earlier frames adds much more viewing baseline, but only for things that stayed put. Because we compute both matches, the disagreement between them is itself a geometric signal of motion. It needs no segmentation network. We use that one signal three ways. It decides which pixels may borrow extra views from the past. It decides which Gaussians enter a long-lived background memory. It also triggers eviction of memory entries that the current cameras can see through. The result is a single-pass, streaming reconstructor for sparse, wide-baseline, dynamic multi-camera video. The file identifies this regime as unaddressed (Gap D), between monocular dynamic methods that fail off-trajectory and static multi-view methods that fail on motion.

Core Claim

For a synchronized sparse rig, a space-time cost volume gives three things at once: better static geometry (from temporal baseline), faithful dynamic geometry (from synchronous-only matching), and a motion mask. The cost volume is a plane sweep that scores candidate depths separately against same-instant views and against pose-warped past views of the same cameras. The mask comes out of the cost volume itself, so it costs little latency and fails in predictable ways.

Concretely, let $C^{s}_p(d)$ be the matching score at pixel $p$ and depth candidate $d$ computed only from other cameras at the current time. Let $C^{\tau}_p(d)$ be the score computed from the rig's frames at earlier times, warped with known or estimated camera poses. Let $d^{*}_p = \arg\max_d C^{s}_p(d)$ be the synchronous depth. The static evidence is

$$ e_p = \sigma\!\left(\beta\,[\,C^{\tau}_p(d^{*}_p) - \max_{d} C^{s}_p(d)\,] + \gamma\right), $$

where $\sigma$ is the logistic function and $\beta,\gamma$ are learned scalars. A static point matches its own past at the synchronous depth, so $e_p$ is high. A moving point does not, so $e_p$ is low. The network fuses the two volumes with a per-pixel gate driven by $e_p$. Static pixels use the sharper combined volume. Dynamic pixels fall back to the synchronous volume alone. Opacity stays tied to the confidence of whichever volume is used, which preserves MVSplat's floater protection. In practice $e_p$ enters as a feature to a small refinement network rather than being hard-thresholded. The equation defines the inductive bias, not a fixed rule.

Research Program

The architecture. We build on MVSplat's backbone: cross-view transformer features and a plane-sweep cost volume, with Gaussian centers taken from predicted depth and opacity taken from matching confidence. We extend it in three ways.

The first extension adds a temporal branch. It sweeps the same depth planes against the rig's previous $K$ frames, warped by rig ego-motion. On driving data the poses are the dataset's calibrated poses. On static rigs they are identity. The two volumes are fused by the static-evidence gate.

The second extension adds a background memory in the spirit of UniSplat's streaming bank. Gaussians enter the memory only after accumulating high static evidence over several frames.

The third extension replaces UniSplat's one-way commit with visibility-based eviction. On every new frame, each memory Gaussian is projected into the current synchronous depth maps. If a current camera confidently sees a surface behind the Gaussian, so that the ray passes through the Gaussian's location, the Gaussian's persistence score drops, and it is removed after a short hysteresis. This directly targets the failure UniSplat documents: a pedestrian misclassified as static leaves a permanent ghost trail. The current cameras then act as ground truth for "what is empty now." No per-frame optimization is involved. Moving objects are drawn from the current frame's synchronous Gaussians. For in-between timestamps we reuse a simple flow-based interpolation of those Gaussians, and we do not claim novelty there.

Training. Training uses rendering losses on held-out cameras at the same timestamp. The data mixes multi-view dynamic sources: subsampled camera rigs from N3DV and DNA-Rendering, and Waymo front and side cameras, which partially overlap. Static RealEstate10K clips are included so the model does not lose static quality. We deliberately add no supervision on the mask $e_p$. One question the study answers is whether a geometric motion signal emerges from rendering losses alone. A variant with weak supervision from optical-flow residuals serves as an ablation.

Analysis. Beyond headline image quality, we measure the fraction of pixels routed to the temporal volume. We compare $e_p$ against ground-truth dynamic masks where they exist. We also run a controlled synthetic study. In it, known-motion objects are composited over RealEstate10K backgrounds, following DGGS's synthetic distractor protocol. We vary object speed, how many cameras overlap, and whether the object stops and restarts. The stop-restart case is the hardest for eviction: an object that pauses looks static and will be committed, so the question is whether eviction clears it once it moves.

Baselines

The naive reference is MVSplat run independently at each timestep on the synchronous views only, retrained on our data mix with the same backbone, resolution, and training budget. It is dynamic-safe by construction and is the most important competitor. If synchronous views alone give enough baseline, the temporal branch adds nothing, and the proposal fails. Beating it means a clear gain in novel-view PSNR and LPIPS on static regions at two input cameras, with no loss on dynamic regions. We report static and dynamic regions separately, using ground-truth or flow-derived masks.

The strongest static-only competitor is DGGS. It finds distractors by cross-view reprojection consistency, refines the masks with a 2D segmenter, and prunes the flagged Gaussians. On the RobustNeRF Statue scene it reports 21.02 dB PSNR against MVSplat's 15.45 dB. We run DGGS's released pipeline on the same inputs and compare only on background reconstruction, since DGGS deletes dynamics. Beating it means matching its background PSNR while also reconstructing the moving content, at lower latency because we run no segmenter.

Among dynamic feed-forward driving methods, the reference numbers on Waymo are DrivingRecon at 27.52 dB PSNR, and UniSplat at 28.56 dB for multi-view reconstruction and 25.12 dB for novel views. UniSplat is the strongest existing method on our question, since it also has a static memory. We evaluate on its protocol and additionally report a ghosting metric: residual error in regions that dynamic objects vacated, measured over the following 30 frames. On that metric, eviction should show its value even if headline PSNR is similar.

Optimization-based ceiling. On N3DV, Instant Gaussian Stream reports 34.15 dB PSNR at 2.67 s per frame, using its full 20-camera rig. We rerun it with our sparse camera subsets as a quality ceiling that includes keyframe optimization. We do not expect to beat it. Success means landing within a stated gap at a fraction of its per-frame time.

Ablations isolate the key ingredient. We compare temporal fusion without the gate, which naively adds past frames to the cost volume, against the gated version. We compare memory with and without visibility eviction. We compare a variant that replaces $e_p$ with SAM-style semantic masks for vehicle and person classes. The ungated variant tests whether motion awareness matters. The semantic-mask variant tests whether a geometric mask beats a semantic one.

Why not just…?

Why not just mask moving objects with an off-the-shelf 2D segmenter and feed MVSplat the accumulated frames? Semantic masks answer "is this a car?" rather than "did this move?" Parked cars would lose their temporal baseline, and unlisted movers would corrupt the cost volume. Split4D reports that 2D-tracking-reliant decomposition (SA4D) reaches 0.593 mIoU against 0.893 for native 4D features. That suggests flickering masks are a real failure source. This alternative is still a real threat on driving data, where the movers are mostly cars and people. The semantic-mask ablation, run at equal latency, settles it directly. If it matches the geometric gate, we will report that honestly.

Why not just unproject monocular depth, as StreamSplat and UniSplat do? Single-image depth has per-frame scale ambiguity. Forge4D needs a dedicated metric-scale module to stop flicker, and StreamSplat reports degradation wherever its depth network fails. Monocular depth also gives no confidence to tie opacity to, which is exactly what suppresses floaters in MVSplat. We will add a variant that feeds a monocular depth prior into our cost volume. If it wins on driving scenes with poor camera overlap, that marks where our approach stops helping.

Why not just optimize a regularized 4D Gaussian model per scene? Quality may be higher, but latency is hours rather than tens of milliseconds, according to the comparisons StreamSplat reports. Under sparse views, regularization alone plateaus low: RegNeRF reaches 18.89 dB on 3-view DTU. We include the IGS ceiling so readers can see the quality-for-speed trade explicitly.

Proposed Experiments

The first decisive check is on N3DV and DNA-Rendering with two and three input cameras, evaluated on held-out cameras. The gated space-time model should beat per-timestep MVSplat on static regions by a clear margin while staying within noise on dynamic regions. The ungated variant should degrade on dynamic regions. If gated and per-timestep results are equal, the core claim is false.

The second check is on Waymo under UniSplat's protocol, where we report PSNR against the 25.12 dB and 28.56 dB references, plus the vacated-region ghosting metric. Eviction should cut ghosting relative to commit-only memory, especially on the stop-and-go pedestrian subset.

The third check covers static degeneracy. On RealEstate10K, the model should not fall meaningfully below MVSplat's reported 26.39 dB. On RobustNeRF, where distractors move between unsynchronized shots, we set the synchronous branch to the single frame and compare against DGGS's 21.02 dB on Statue. No reported result was found on Lacuna for this exact configuration.

For latency, we measure end-to-end per-frame reconstruction time on an A100 as a function of the number $K$ of past frames. The target stays within a small multiple of MVSplat's reported 44 ms, so the method remains interactive.

Finally, the synthetic sweep over object speed and pausing should show the failure boundary of the gate. Slow objects are expected to be misread as static until they move a pixel-scale distance, and the sweep should show whether eviction recovers them within a bounded number of frames.