Recent networks can turn a handful of unposed photos into a 3D scene in under half a second. They do this by predicting thousands of small colored 3D blobs, called 3D Gaussians, in a single forward pass of a neural network, with no per-scene optimization. A graphics card can then draw those blobs at more than a hundred frames per second. The weak point is the camera poses: the networks must also guess where each photo was taken from. Explicit blobs are unforgiving here. A small pose error puts every blob in the wrong place, which produces doubled edges and floating debris (PF3plat). Moving objects make it worse, because a walking person looks like a geometric contradiction that pulls the pose estimate off. The classical fix is to refine each pose by rendering the scene and comparing the render against the photo. That only works when the starting guess already lies inside a narrow "basin of attraction," the region of starting points from which gradient steps lead to the right answer. Moving objects add false minima to this landscape.
We propose to train the reconstruction network so that its own output can serve as a well-behaved surface for pose refinement. Alongside color, every predicted Gaussian carries a learned feature vector and a "static" weight. The features are trained with an explicit objective: a few Gauss-Newton steps on the rendered-feature residual, starting from a perturbed pose, must land back on the true pose. The static weight removes moving objects from that residual. Refinement then costs a handful of rasterization passes at low resolution, which is milliseconds, rather than seconds of optimization. In two sentences: the network learns to emit a scene whose feature renders have a wide, distractor-free basin of attraction, so its own rough pose guesses can be fixed almost for free. That fix matters most when moving objects corrupt the first guess.
Core Claim
Feed-forward pose estimation in dynamic scenes has already improved a great deal. PAGE-4D adapts the VGGT geometry model so that its pose tokens ignore moving regions, cutting Sintel camera trajectory error from 0.214 to 0.143. Even so, it outputs point maps rather than renderable Gaussians, and its remaining pose error is the kind that explicit Gaussians amplify. Our claim is that the remaining gap should be closed by fast, learned, test-time refinement rather than by a better one-shot regressor. We further claim that the refinement surface itself should be shaped during training.
Concretely, let $\xi$ be a camera pose and $F_p(\mathcal{G}, \xi)$ the rendered feature at pixel $p$ from predicted Gaussians $\mathcal{G}$. Let $\Phi_p(I)$ be the feature the network extracts from the observed image $I$ at pixel $p$, and $w_p$ the rendered static weight. Refinement minimizes
$$L(\xi) = \sum_p w_p \,\lVert F_p(\mathcal{G}, \xi) - \Phi_p(I) \rVert^2 .$$
During training we sample perturbations $\delta$ of growing magnitude around the true pose $\xi^\ast$, unroll $K$ Gauss-Newton steps $\mathrm{GN}_K$, and penalize
$$\mathcal{L}_{\text{basin}} = \mathbb{E}_{\delta}\,\lVert \mathrm{GN}_K(\xi^\ast + \delta) \ominus \xi^\ast \rVert ,$$
where $\ominus$ is the pose difference on the rotation-translation group. This loss directly rewards a wide basin, and it rewards static weights that suppress any region whose residual pulls the solver away from $\xi^\ast$.
Prior work handles each piece separately. Basin widening has been done with fixed, hand-chosen tools. NeMo uses contrastive features on known meshes. AlignPose renders frozen DINOv2 features from CAD models. LATITUDE low-pass filters a per-scene NeRF. In none of these is the representation learned to make refinement succeed, and none handles moving objects. Fast inversion, as in iGaussian, needs a pre-optimized static map. GGRt refines poses iteratively from image features, not from renders of its own Gaussians, and assumes a static scene.
Research Program
The system has three parts. The first is a pose-free Gaussian predictor. We start from a PF3plat-style coarse-to-fine design and replace its pose stage with PAGE-4D-style masked attention, giving coarse poses in the presence of motion. The second is a feature-and-weight head that attaches a low-dimensional feature and a static weight $w$ to every Gaussian. The third is a differentiable $K$-step Gauss-Newton refiner. It rasterizes feature maps at reduced resolution and runs at inference before a final pass of the Gaussian head, which re-places the Gaussians using the refined poses. The static weights also act as a single-pass pruning signal. Gaussians with low $w$ are dropped from the static map, which targets the floater debris that SpotlessSplats removes with multi-pass pruning based on how much each Gaussian contributes, and that DG-SLAM removes with per-keyframe optimization.
For training data, we build controllable motion by compositing moving objects onto RealEstate10K and ACID backgrounds, following the synthetic distractor protocol of DGGS. This gives exact poses and masks. We add the dynamic video data used to fine-tune PAGE-4D. The static weights get no mask supervision in the main model, so any motion segmentation must come from the basin loss and the rendering loss alone. A variant with mask supervision is reported separately.
The analysis has two parts. First, we measure basin width directly. We perturb ground-truth poses by controlled rotations and translations and record how often refinement converges, as a function of initial error and of the fraction of the image covered by moving objects. We do this for learned features, frozen DINOv2 features, and RGB. Second, we measure end-to-end latency: reconstruction time, refinement time, and rendering frames per second.
Baselines
The naive reference point is a static pose-free predictor. PF3plat is run unchanged on the dynamic and distractor benchmarks, so we can measure how much motion actually hurts. On static RealEstate10K, PF3plat reports 23.589 dB PSNR. Our model must stay within noise of that number when nothing moves.
The strongest existing competitor is a fully feed-forward cascade. PAGE-4D supplies poses from dynamic video. Those poses are passed to DGGS, which needs poses and then removes distractors and outputs Gaussians in one pass. This cascade is the strongest competitor because each half is the best documented feed-forward component for its sub-problem, and it needs no test-time optimization. For a fair comparison, both components will be retrained on our training data with matched GPU budget and tuning sweeps, and evaluated on identical frame sets. On the RobustNeRF "Statue" scene, DGGS reports 21.02 dB PSNR (versus 15.45 dB for MVSplat) given known poses. That is an upper reference for the cascade. On Sintel, PAGE-4D reports ATE 0.143. We count the proposal as beating the cascade if three things hold together: it lowers Sintel and Bonn trajectory error, it lowers the rate of large pose errors in the perturbation study, and it improves static-background PSNR on NeRF On-the-Go and RobustNeRF scenes. The total added latency for these gains must be under the cost of one forward pass.
The optimization-based reference is DG-SLAM. It reports 5.5 cm average ATE on Bonn at about 2 FPS. It uses RGB-D input, so it is a ceiling to approach rather than a like-for-like competitor, and we will report it as such.
The key ablations isolate the main ingredient. We will run the identical refiner on frozen DINOv2 features with no basin loss, and on RGB with a coarse-to-fine blur schedule. We will also run the full model with $w \equiv 1$, which isolates the role of motion suppression in the refinement objective.
Why not just…?
Why not just chain PAGE-4D poses into DGGS? This is the real threat, and it is the main baseline. If the cascade's trajectory error is already low enough that Gaussian misalignment does not show in renders, our refinement adds nothing, and the honest conclusion would be that one-shot regression suffices. The evidence that it might not suffice is indirect. PF3plat documents that explicit Gaussians are very sensitive to small center errors, and PAGE-4D's residual 0.143 ATE on Sintel is far from zero. The perturbation study settles the question. If refinement starting from cascade poses yields no measurable PSNR or ATE gain, we drop the claim.
Why not just refine with frozen DINOv2 features, as AlignPose does? AlignPose shows that DINOv2 features give wide basins for objects with known CAD models and calibrated cameras. Nothing in that work tests features rendered from noisy, predicted Gaussians with people walking through the frame. In that setting, foundation features of moving objects are perfectly well-formed features and will pull the solver toward false minima. The frozen-feature ablation answers this directly. If it matches the learned features, the basin loss is unnecessary, and we would report the simpler system.
Why not just blur coarse-to-fine on RGB, as LATITUDE does? LATITUDE widened the convergence region to 12 m and 12° on city-scale NeRFs. However, it needed a per-scene model and a careful lower frequency cutoff, because fully blurred renders lose all features. Blurring also does nothing about moving objects, which stay large-scale structure at every blur level. The RGB-schedule ablation measures whether this cheaper approach closes the gap.
Why not just mask people and cars with a segmentation model first? The source evidence suggests three problems. Off-the-shelf segmenters miss non-standard movers such as boxes being carried or balloons on Bonn. DG-SLAM found semantic priors needed geometric verification. DGGS found that external mask refinement adds latency. We will include a segmentation-masked PF3plat variant to show where class-based masking fails.
Proposed Experiments
Pose accuracy will be measured as ATE on Sintel and as ATE and RPE on TUM RGB-D and Bonn walking and box-carrying sequences. The comparison is against PAGE-4D (Sintel ATE 0.143), the retrained cascade, and the ablations, with DG-SLAM's 5.5 cm on Bonn as an RGB-D reference. Static-background synthesis will be measured as PSNR, SSIM, and LPIPS on RobustNeRF scenes, NeRF On-the-Go, and the DGGS-style synthetic distractor splits of RealEstate10K and ACID, for which no reported number was found on Lacuna. DGGS's 21.02 dB on Statue and SpotlessSplats' per-scene optimized 22.81 dB on Fountain serve as references. Static non-regression is checked on RealEstate10K against PF3plat's 23.589 dB.
Latency is reported as reconstruction milliseconds, refinement milliseconds for each $K$, rendering frames per second, and Gaussian count after pruning by static weight.
The decisive check is the basin curve. It plots refinement success rate against initial pose error, separately for clean and heavily dynamic frames. The proposal succeeds only if the learned, motion-weighted features keep convergence high at error levels where the RGB and DINOv2 refiners collapse, and if the gain carries through to rendering quality beyond the PAGE-4D-plus-DGGS cascade.
References
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting for Novel View Synthesis
- PAGE-4D: Disentangled Pose and Geometry Estimation for VGGT-4D Perception
- NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation
- AlignPose: Generalizable 6D Pose Estimation via Multi-view Feature-metric Alignment
- LATITUDE: Robotic Global Localization with Truncated Dynamic Low-pass Filter in City-scale NeRF
- iGaussian: Real-Time Camera Pose Estimation via Feed-Forward 3D Gaussian Splatting Inversion
- GGRt: Towards Pose-Free Generalizable 3D Gaussian Splatting in Real-Time
- SpotLessSplats: Ignoring Distractors in 3D Gaussian Splatting
- DG-SLAM: Robust Dynamic Gaussian Splatting SLAM with Hybrid Pose Optimization
- Distractor-free Generalizable 3D Gaussian Splatting