Feed-forward 3D reconstruction networks turn a handful of photos into a renderable 3D scene in a fraction of a second. They break when something in the scene moves. A walking person gives two images contradictory depths for the same pixels, and the network fills the space with floating blobs. The current fixes come from outside the network. One masks moving objects with a pretrained segmenter. Another computes camera poses offline with a separate SLAM system (simultaneous localization and mapping). A third drops "inconsistent" content using hand-set thresholds. This proposal puts a small, exact geometric solver inside the network instead: a weighted least-squares fit of the camera motion between frames. The per-pixel weights the network feeds that solver then become its definition of "static."
The weights are trained only through the camera-pose error and the rendering error, never with mask labels. Pixels on moving objects corrupt the camera fit, so the network learns to down-weight them. The same weights then split the scene's 3D Gaussians into a static background fused across frames and a dynamic foreground with its own motion. In two sentences: make the camera solver differentiable and weighted, and let the weights it learns to trust become the static/dynamic decomposition of a feed-forward 4D reconstruction. The geometry, rather than a segmenter, decides what is moving.
Core Claim
A closed-form, differentiable weighted rigid alignment placed between a feed-forward depth-and-correspondence backbone and a 3D Gaussian Splatting head does three things at once. It recovers camera motion that is robust to independently moving content, without external SLAM. It produces static/dynamic masks that match ground-truth motion masks without ever seeing mask labels. It keeps rendering quality on fully static scenes. This last point matters because fixed-trimming robust losses such as RobustNeRF are reported to lose quality on clean captures. The specific claim we test is that the solver is the essential ingredient. A pose-regression head plus a mask head of equal capacity, trained on the same losses, should yield worse masks and worse static-region rendering as the fraction of moving pixels grows.
Where This Sits
The method borrows from three works that show the mechanism is sound in isolation. SC-wLS learns per-correspondence weights for a differentiable weighted least-squares camera solver, supervised only by pose. It assumes a static world and outputs no scene. Discovery of Latent 3D Keypoints shows that a differentiable SVD-based Procrustes layer (the classic closed-form fit of a rotation and translation between two point sets) can train upstream features end to end. SpatialTrackerV2 comes closest in spirit. It predicts per-point dynamic probabilities that weight residuals in a differentiable bundle adjustment, but it outputs tracks and poses, not a renderable scene. We reuse the weighted-solver idea from all three. The new step is closing the loop through a renderer, so the weights are also shaped by whether the resulting static/dynamic split renders well.
The works that motivate the gap are the feed-forward reconstructors. PF3plat does pose-free feed-forward splatting by refining poses with learned transformer offsets. It states plainly that it handles static scenes only. BTimer reconstructs dynamic "bullet-time" snapshots in about 150 ms, but it needs DROID-SLAM poses and Segment Anything masks to train, and it reports geometry that looks right but is inaccurate. Any4D and 4RC regress poses and scene flow purely through attention, with no analytical constraint and no view-synthesis head. Any4D also anchors motion to the first frame.
Distractor-free Generalizable 3DGS is the closest feed-forward competitor. It flags transient content by reprojection consistency, refines the masks with a 2D segmenter, and prunes the flagged Gaussians, so it discards the moving content rather than reconstructing it. Our proposal differs from it on both counts: the masks come from a learned solver and not from heuristics plus segmentation, and the dynamic part is kept and rendered.
Two works constrain the design. PF3plat reports that Gaussian centroids give unstable gradients unless geometry is warm-started, so we initialize from a coarse depth prior. The research file notes that SVD and normal-equation layers produce exploding gradients when the system becomes ill-conditioned. Collapsing weights is exactly how our solver could become ill-conditioned, so we regularize against it.
Research Program
The model takes a short monocular clip of $T$ frames with known intrinsics. A shared backbone predicts four things: a per-frame depth map, dense correspondences between adjacent frames, a per-pixel weight $w_i \in [0,1]$, and per-pixel Gaussian parameters. Each correspondence gives a pair of back-projected 3D points $(p_i, q_i)$ in frames $t$ and $t+1$. The relative camera motion $(R, \tau)$ is the closed-form solution of
$$ (R,\tau) = \arg\min_{R \in SO(3),\, \tau} \sum_i w_i \,\lVert R p_i + \tau - q_i \rVert^2 , $$
which is solved by one weighted SVD and is differentiable in $w_i$, $p_i$ and $q_i$. The poses are chained across the clip. The solver supplies the only pose estimate, so the pose loss reaches the weights directly.
Each pixel then has a residual 3D flow, meaning its observed motion minus the motion implied by the solved camera. Pixels with high weight are lifted to static Gaussians and fused across frames in the solved world frame. Pixels with low weight become dynamic Gaussians, displaced by their residual flow and placed at the query timestamp. A held-out frame is rendered, and a photometric loss flows back through the Gaussians, the solver and the weights. Where ground-truth poses exist, a rotation-and-translation loss is added. No mask supervision is used anywhere.
Two terms keep the solver well-posed. The first is a floor on total weight mass. The second is a penalty that keeps the smallest singular value of the weighted cross-covariance matrix bounded away from zero, which stops the weights from collapsing onto a degenerate subset. During training we monitor gradient norms through the SVD to check that these terms are enough.
Training starts from static data, on the observation in BTimer that a static scene is a dynamic scene with zero motion. It then moves to two dynamic sources. The first is synthetic composites built on RealEstate10K backgrounds with pasted moving objects, the protocol DGGS uses for its distractors. Here the masks and poses are known exactly. The second is real dynamic video from Stereo4D, the dataset SpatialTrackerV2 trains on, using its pose annotations.
Baselines
The naive reference point is a static feed-forward splatter applied directly to dynamic clips: PF3plat as released. It shows how badly the static assumption fails and sets the bar on clean static data. There it reports PSNR 23.589 on RealEstate10K with 0.39 s reconstruction. We run the released model on our exact splits and quote our own measurement next to that number. Beating it on static data is not required. Matching it within noise is, since our claim includes no clean-data penalty.
The strongest existing feed-forward method for clean backgrounds under motion is DGGS. We retrain it on our composites with the same backbone scale, data and iteration budget, and compare static-region PSNR and mask IoU. For dynamic rendering we compare to BTimer, giving it its required DROID-SLAM poses and SAM masks. Our method gets neither, which favors BTimer. For pose and geometry we compare to Any4D and SpatialTrackerV2 on Stereo4D camera-trajectory error. The research file contains no dynamic-scene numbers for any of these, so we report none from the literature: no reported result found on Lacuna for BTimer, DGGS, Any4D or SpatialTrackerV2 on Stereo4D or on composited RealEstate10K. All of them are rerun.
Beating DGGS means higher static-region PSNR on the composites at a matched number of input views. It also means our unsupervised masks reach at least DGGS's segmenter-assisted mask IoU.
The key ablation swaps the solver for a regression head of matching parameter count that outputs $(R,\tau)$, plus a separate mask head. Both are trained with the identical rendering and pose losses. If this ablation matches the full model across all levels of motion, the claim fails.
A second ablation keeps the solver but stops gradients from the rendering loss into the weights. It tells us whether the renderer adds anything beyond pose supervision.
Why not just…?
Why not just regress the pose with a transformer and add a motion-mask head, as Any4D and 4RC effectively do? This is the real threat, and it is exactly our key ablation. The argument for the solver is structural. In a regressor, a mask head receives gradient only from rendering, and rendering can be satisfied by explaining motion with depth errors, which is the inaccurate geometry BTimer reports. In the solver, a mislabeled moving pixel directly corrupts the camera fit, so pose loss pushes its weight down. We expect the gap to be small at low motion fractions and to widen as moving pixels grow toward a large minority. If it never widens, we will report that and drop the claim.
Why not just mask movers with a pretrained segmenter and run a static model? Segmenters label what can move, not what is moving. A parked car and a driving car look the same to them. Split4D is motivated precisely by segmentation errors propagating into 3D. Our composites include static instances of movable categories next to moving ones, so mask IoU against true motion settles the question directly.
Why not just run RANSAC on the predicted correspondences at test time? RANSAC (random sample consensus) is robust at inference, but it is not differentiable. The backbone would then never learn depth and correspondences suited to separating motion, which is the decoupling problem Single-Stage 6D Object Pose Estimation describes. We include a RANSAC-at-test-time variant of the regression ablation so this comparison is measured, not assumed.
Proposed Experiments
The decisive experiment is a sweep over the fraction of moving pixels on composited RealEstate10K, from 0% to about 60%. For each method we record static-region PSNR, rotation and translation error against the known trajectory, and IoU between the thresholded weights and the true motion mask. We predict that the solver model's pose error stays flat while static pixels remain the dominant coherent motion. We predict that the regression ablation degrades earlier. We also predict a characteristic failure when a single large object moving rigidly outweighs the background, because the weighted fit can then lock onto it. We will report this failure honestly, since it marks the method's boundary.
On clean RealEstate10K we check that PSNR stays within noise of our rerun of PF3plat. On Stereo4D we report trajectory error against Any4D and SpatialTrackerV2. We also report novel-view PSNR on dynamic regions against BTimer, run with its privileged poses and masks.
Timing is measured on one GPU against the reported figures: 0.39 s for PF3plat, about 150 ms for BTimer, and 0.5 s for 50 frames on an H100 for Any4D. The extra SVD per frame pair should add negligible cost.
The result that would most change our view is mask IoU. If unsupervised solver weights reach segmenter-assisted IoU and separate parked from moving instances, the solver is doing work no prior method does.
References
- RobustNeRF: Ignoring Distractors with Robust Losses
- SC-wLS: Towards Interpretable Feed-forward Camera Re-localization
- Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
- SpatialTrackerV2: 3D Point Tracking Made Easy
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting for Novel View Synthesis
- Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos
- Any4D: Unified Feed-Forward Metric 4D Reconstruction
- 4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere
- Distractor-free Generalizable 3D Gaussian Splatting
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Single-Stage 6D Object Pose Estimation