Pose Priors as a Noisy Modality in Feed-Forward Gaussian Splatting

Feed-forward 3D reconstruction networks turn a handful of photos into a 3D scene in one pass of a neural network, taking milliseconds rather than the minutes of per-scene optimization. Current systems fall into two camps. Posed models are given the camera positions and use them to triangulate depth. Pose-free models ignore camera positions and infer everything from pixels. Real devices rarely fit either camp. Phones, robots and drones report camera poses from inertial sensors, odometry or tracking systems, and those poses are usually useful but sometimes wrong. A posed model that receives a drifted pose projects features to the wrong places and breaks. A pose-free model throws away a signal that would have fixed its scale and depth. We propose to treat the incoming pose the way sensor-fusion work treats a LiDAR or a second camera: as a separate input stream whose reliability must be checked against the images before it is trusted.

Concretely, we build a feed-forward Gaussian splatting model that accepts optional, possibly corrupted camera poses. The model estimates, for each view, how far the supplied pose agrees with dense image-feature correspondences, and uses that agreement to decide how strongly the pose steers reconstruction. Gaussian splatting here means representing the scene as many small colored 3D ellipsoids that can be rendered very quickly. The target behavior is simple to state and, as far as the research file shows, not demonstrated by any existing system. As input pose noise grows from zero to large, reconstruction quality should fall smoothly from the quality of a posed model to the quality of a pose-free model and never drop below it. At intermediate noise it should beat both.

Core Claim

The literature frames a methodological disagreement. pixelSplat builds its attention around epipolar lines, the lines in one image along which a pixel from another image must lie given known poses, and argues this geometry is needed to resolve depth and scale. NoPoSplat shows that a plain Vision Transformer without any pose input outperforms pose-requiring methods when views overlap only 5–30%. eFreeSplat drops epipolar sampling in favor of a pretrained cross-view transformer, but still needs relative poses for its warping and degrades when views are very close together. Our claim is that this is largely a disagreement about pose reliability rather than architecture. Explicit geometry wins when poses are right, and implicit matching wins when poses are absent or wrong. A model that measures pose reliability from image evidence, per view and per scene, should inherit the advantages of both regimes.

The closest existing system is MapAnything. It ingests heterogeneous optional inputs, including RGB, intrinsics, coarse poses and depth, through DINOv2 encoders and a multi-view transformer, and predicts metric geometry and poses. The research file notes three limits: it fuses these inputs by concatenation and transformer layers, does not explicitly model noise or uncertainty in the supplied poses, and outputs no explicit Gaussian primitives for real-time rendering. PF3plat refines poses coarse-to-fine from frozen depth and matching networks but uses heuristic confidence thresholds and never consumes an external pose prior. Our proposal differs from both in three ways. It trains on a calibrated distribution of corrupted poses. It computes an explicit agreement signal between the pose stream and the image stream. It is evaluated on a full pose-noise sweep rather than at the two endpoints.

Mechanism

We start from a NoPoSplat-style backbone because it is simple and fast. That backbone is a ViT encoder-decoder that predicts Gaussians in the coordinate frame of the first image, with intrinsics injected as tokens, reported at 15 ms for two 256×256 images. For each non-reference view $i$ we optionally receive a pose prior $P_i$. We compute dense DINOv2 feature matches between view $i$ and the reference view, and measure how well those matches satisfy the epipolar constraint implied by $P_i$. The measure is the median Sampson residual $r_i$, the standard first-order distance of a match from its epipolar line. A small network maps this residual to a per-view reliability weight

$$w_i = \sigma\big(g(r_i, c_i)\big),$$

where $\sigma$ is the logistic function, $g$ is a small learned network, and $c_i$ is the fraction of matches that pass a mutual-nearest-neighbor check. This fraction tells the model whether the residual itself can be believed, for example in textureless rooms.

The pose then enters the backbone through cross-attention. Image tokens query ray-embedding tokens computed from $P_i$, and the attention output is scaled by $w_i$. The fusion pattern comes from SGFormer, where 3D geometric tokens attend to 2D image features and stay robust under sensor dropout and reduced field of view. We reverse the direction, and our gate is computed from measured cross-modal disagreement rather than learned implicitly. The model also outputs a refined pose, obtained by Perspective-n-Point on the predicted Gaussian centers as in NoPoSplat, so the prior is corrected rather than merely accepted or discarded.

Training uses RealEstate10K and DL3DV with the photometric rendering loss plus a pose loss. We corrupt ground-truth poses with a mixture of three kinds of error: small Gaussian jitter, large rotation and translation outliers applied to a random subset of views, and smooth scale and trajectory drift of the kind monocular tracking produces. We also drop poses entirely for some training samples, so the pose-free regime is always covered.

Research Program

The work has three parts. First, we measure the problem. We run a posed feed-forward model (eFreeSplat), a pose-free model (NoPoSplat) and MapAnything on RealEstate10K, ACID and ScanNet++ across a sweep of injected pose noise. This produces the quality-versus-noise curves that, to our knowledge, nobody has published, and it locates the crossover noise level at which ignoring poses becomes better than using them. Second, we build the gated model described above and train it on the corruption mixture. Third, we test whether the benefit carries over to the cross-modal task that motivates it. EmbodiedSplat lifts CLIP features onto feed-forward Gaussians online and reports 46.22% mIoU on ScanNet 19-class segmentation, but it requires known camera poses. The research file lists calibration drift as a core weakness of projecting 2D semantics onto 3D points. We feed EmbodiedSplat's semantic-lifting stage with three pose sources: ground truth, poses from a real monocular tracker, and poses corrected by our model. We then measure how much of the mIoU lost to drift our corrections recover. For the realistic tracker poses we use Flash-Mono, a feed-forward Gaussian SLAM system evaluated on ScanNet, so that the noise is not only synthetic.

Baselines

The naive reference point is a posed feed-forward model that consumes the corrupted poses as if they were correct. We use eFreeSplat, which reports 26.45 dB PSNR on RealEstate10K and 28.30 dB on ACID with clean poses, retrained under our resolution and split. This baseline shows how badly trusting a bad prior hurts. Its clean-pose number also sets the ceiling that our model should approach at zero noise.

The pose-free reference is NoPoSplat, retrained on the same data mixture with identical backbone size, resolution and training steps. It ignores the prior entirely, so its curve is flat across the noise sweep. Our model counts as beating it only if we exceed it at every noise level short of pure adversarial corruption, and never fall more than 0.2 dB below it in any condition. We also report PF3plat's published 23.589 PSNR on RealEstate10K as a published pose-free reference point, noting that it was measured under PF3plat's own protocol.

The strongest existing competitor for this specific question is MapAnything, because it already accepts optional coarse poses alongside images. It reports relative errors of 0.01 for points and 0.06 for poses on multi-view reconstruction when conditioned on images, calibration, poses and depth. Its training scale across 13 datasets cannot be matched fairly. We therefore compare in two ways. We run the released model as-is on the noise sweep for pose accuracy and depth, since it outputs no Gaussians. We also build a MapAnything-style variant inside our own backbone, with pose tokens concatenated and no gate, trained on exactly our data and schedule. Beating the in-house variant at matched compute is the fair test of our design. Beating or matching the released model on pose error at high noise is a stronger but secondary claim.

The key-ingredient ablation isolates the gate and its input. One version trains the ungated concatenation variant on the same corruption mixture, which tests whether noise augmentation alone accounts for any gain. A second computes the gate from photometric reprojection error instead of DINOv2 feature residuals, which tests whether cross-modal feature agreement is what matters. The two gated versions and the ungated version share architecture, data and tuning budget, with the same number of hyperparameter trials each.

Why not just…?

Why not just run a pose estimator, or trust the sensor, and feed the result to a posed model? This is the naive baseline above. NoPoSplat already documents the cascading errors of estimate-then-reconstruct pipelines and the failure of COLMAP at low overlap, and PF3plat reports that small pose misalignments destabilize pixel-aligned Gaussians. Our noise sweep will show directly where this pipeline falls below the pose-free curve. We expect this to happen at modest noise, and the result will be reported whichever way it falls.

Why not just train a pose-conditioned model with noise augmentation and let attention sort it out? This is the most serious threat. A transformer trained on corrupted poses may learn to down-weight them without any explicit residual. The ungated ablation settles it. If the ungated variant matches the gated model across the sweep, including the per-view outlier condition, then the explicit agreement signal adds nothing. In that case the contribution shrinks to the benchmark and training recipe, and we will say so. We expect the gate to matter most when one view's pose is badly wrong and the others are fine, because concatenation has no natural per-view off-switch.

Why not just predict pose-free and then fuse the prediction with the prior afterward, for example by weighted averaging of poses or aligning scale to the prior? This late fusion is cheap and can fix pose error. It cannot improve the Gaussians, however, because they were triangulated without the baseline information the prior carries. We implement it on top of NoPoSplat with the averaging weight tuned on validation data. If it matches our model on rendering quality and not only on pose error, then feeding the prior into reconstruction is unnecessary. We would then redirect the work toward the semantic-lifting application, where corrected poses alone may be what matters.

Why not just optimize poses and Gaussians per scene at test time? PF3plat reports that InstantSplat takes 53 seconds per scene against 0.39 seconds for feed-forward reconstruction. iGaussian reports that render-and-compare loops drop to 0% success at low overlap, where its feed-forward approach reaches about 82%. We include a short test-time refinement run, initialized from the corrupted prior, as an accuracy upper reference and report it with its wall-clock time. We do not claim to beat it on quality at zero latency cost.

Proposed Experiments

The primary experiment is the noise sweep on RealEstate10K, ACID and ScanNet++. We use five levels of rotation and translation noise plus a no-pose condition, together with two structured conditions. The first gives one view a gross pose outlier. The second applies smooth drift taken from Flash-Mono trajectories on ScanNet. We report PSNR, SSIM, LPIPS, and the rotation and translation error of the refined poses. The decisive check has two parts. First, at every level our curve should lie at or above the maximum of the posed and pose-free curves, within noise across three seeds, and strictly above both near their crossover. Second, in the outlier condition the learned weight $w_i$ should separate corrupted from clean views, which we measure with the area under the ROC curve over views.

A second check tests generalization of the gate. We train on synthetic corruption only and evaluate on Flash-Mono drift. If the gate fails there, it has learned the synthetic noise statistics rather than genuine disagreement between pose and images.

The third experiment is the downstream ScanNet semantic segmentation test using EmbodiedSplat's lifting stage. The reference is its reported 46.22% mIoU with known poses. The measured quantity is the fraction of the mIoU drop under tracker poses that our corrected poses recover, compared against the recovery from late fusion.

A negative result would still be informative. If feed-forward models are insensitive to realistic pose noise, then using poses and going pose-free are equivalent in practice. The case for building pose-conditioned feed-forward systems at all would then weaken considerably.