Systems that build a 3D scene from a handful of unposed photos in under a second now exist. They first estimate where each camera was, then predict a cloud of small colored 3D blobs that can be rendered from new viewpoints; this representation is called 3D Gaussian splatting. These "feed-forward" systems are fast because a single network pass replaces minutes of per-scene fitting. They are also fragile in a specific way. Their camera estimates rest on matched pixels between images ("correspondences"), and on repetitive facades, mirrors, or textureless walls most of those matches can be wrong. The slower methods that jointly fit cameras and geometry by gradient descent are more accurate, but they need ten minutes to an hour per scene and still get stuck when the starting cameras are badly off.
We propose to put an explicit geometric test for wrong matches inside a short learned refinement loop, and to repeat that test at every step. At each of a few steps (around four), the system lifts matches into 3D using its current depth estimate. It scores which matches agree with each other about the camera motion, solves for the camera in closed form from the agreeing matches, updates the Gaussians from rendering errors, and then re-lifts the matches with the improved depth. Matches that looked inconsistent only because the early depth was wrong get readmitted. Matches that are consistent but wrong, such as those from repeated windows, stay suppressed because they disagree with the dominant motion. Read in two sentences: feed-forward reconstruction becomes robust to heavily corrupted matches when outlier rejection is part of the iterative pose-and-geometry update rather than a one-time preprocessing filter. That rejection must also be made scale-aware, because depth predicted from images has no fixed metric scale.
Core Claim
Three lines of evidence point to this gap. First, pose-free feed-forward reconstructors handle wrong matches only implicitly. PF3plat builds coarse poses from LightGlue matches and monocular depth, then gates them with a learned cost-volume confidence. AnySplat regresses intrinsics, extrinsics and Gaussians directly with global attention. Neither verifies that matches agree with a single rigid motion. Second, robust point-cloud registration shows that such verification survives extreme contamination. PointDSC builds a graph whose edges mark pairs of matches that preserve 3D distances, reads inliers off the graph's leading eigenvector (a relaxation of the NP-hard maximal-clique search), and reports aligning scans at inlier ratios as low as 1–3%. That work operates only on metric point clouds, never on image-derived geometry or rendering. Third, joint-optimization papers disagree about what should drive pose updates. Look Gauss, No Pose relies on photometric gradients. 3R-GS and GSLoc argue for correspondence geometry instead, but they consume MASt3R matches either unverified (3R-GS's epipolar loss) or through a single RANSAC pass against an already-built scene (GSLoc).
Our claim is that a recurrent refiner should compute pose updates from correspondences whose inlier weights come from a differentiable, scale-decoupled spatial-consistency test, recomputed at every step. Under heavy match corruption, this should deliver the pose accuracy of per-scene joint optimization at near feed-forward latency. Gaussians themselves are updated from rendering error, as in ReSplat, a gradient-free recurrent splatting model. ReSplat assumes ground-truth poses, so it does no pose estimation at all.
Research Program
The system starts from an existing pose-free feed-forward initializer (PF3plat for two views, AnySplat for more), which supplies coarse cameras, per-view depth with confidence, and Gaussians. A standard matcher supplies candidate correspondences. At each refinement step, each match is lifted to a 3D point in each view's own camera frame using the current depth. Rendered depth replaces predicted depth after the first step.
The obstacle to using registration machinery here is that image-derived depth has an unknown scale per view. Rigid distance preservation therefore fails, a limitation the research file notes for visual correspondences. We handle this by decoupling scale before testing consistency, in the spirit of the translation–rotation decoupling in DeTarNet and Liu et al.'s translation search. For a sampled pair of matches $i,j$, let $d_{ij}$ and $d'_{ij}$ be their 3D distances in the two views. For inliers, the log ratio $\log(d_{ij}/d'_{ij})$ concentrates at the log of the true relative scale $s$. Outliers spread out, so a robust mode estimate $\hat{s}$ is cheap to compute. The compatibility between matches is then
$$\beta_{ij} = \max\!\left(0,\; 1 - \frac{(d_{ij} - \hat{s}\, d'_{ij})^2}{\sigma^2}\right),$$
where $\sigma$ is a noise tolerance set from the depth confidence. As in PointDSC, the leading eigenvector of the matrix $[\beta_{ij}]$, obtained by a few power iterations, gives soft inlier weights $w_i$.
The camera update is a weighted closed-form similarity alignment (rotation, translation, scale) of the lifted point sets, which is differentiable in the weights. It is not regressed by a network, so the geometry of the solve is explicit. A small network then predicts residual corrections to the pose and updates to the Gaussians from rendered-versus-input error maps, following ReSplat's design. The whole loop is trained end to end on RealEstate10K and DL3DV with photometric loss and pose supervision. During training, synthetic false matches are injected at random rates and repetitive-texture crops are oversampled, so the learned parts never see only clean matches.
We will also analyze where the gains come from. We will track, per step, how many true inliers are rejected and then readmitted, and test whether readmission correlates with depth error at that step. If re-verification never readmits anything, the "inside the loop" claim is false and the method reduces to one-shot filtering.
Baselines
The naive reference is COLMAP poses fed to a posed feed-forward model, DepthSplat, with all failed registrations counted as failures. We report its wall-clock cost, including the structure-from-motion step, which GS-LRM-style posed models leave out. 3R-GS reports COLMAP rotation errors as high as 133° on ScanNet, so we expect it to fail often in sparse views. It is the right "do nothing new" anchor.
The strongest feed-forward competitors are PF3plat and AnySplat, run from their released weights. The fair comparison retrains both on the same data with the same synthetic-outlier augmentation, with matched parameter counts. PF3plat reports 23.59 dB PSNR on RealEstate10K two-view and above 21 dB zero-shot on DL3DV. AnySplat reports 18.32 dB on six-view Mip-NeRF 360. Beating them means higher PSNR and lower rotation error at matched latency (under one second for two to eight views). The margin should grow as the outlier rate rises. On clean RealEstate10K we expect at most a small gain, and we would treat parity there as acceptable.
The strongest accuracy reference is per-scene joint optimization: 3R-GS initialized from the same coarse poses, plus Look Gauss, No Pose. 3R-GS reports 9.3° rotation error on ScanNet and a PSNR improvement from roughly 20.5 to 24.0 dB on Tanks and Temples. We run 3R-GS both to convergence and truncated to our wall-clock budget. Beating it outright is not expected. The goal is to land closer to converged 3R-GS than truncated 3R-GS does, at one to two orders of magnitude less time.
Three ablations isolate the key ingredient. The first verifies matches once before the loop and then freezes the weights. The second removes the consistency layer and lets attention weight matches. The third uses rigid rather than scale-decoupled compatibility.
Why not just…?
Why not just run RANSAC with a pose solver on MASt3R or LightGlue matches once, then hand the poses to a feed-forward model? This is the most serious threat. On moderately clean matches it may match our accuracy. GSLoc shows that one RANSAC-plus-PnP pass already refines DFNet poses on 7Scenes from 6 cm / 1.93° to 1.1 cm / 0.34°. Its weakness is the high-outlier regime. RANSAC's sample count grows steeply with the outlier ratio, and PointDSC reports being about 60× faster than RANSAC with 100k iterations at higher recall on 3DMatch. A one-shot filter also cannot readmit inliers that early depth errors misjudged. We will run RANSAC-then-feed-forward and our verify-once ablation across injected outlier rates. If neither falls clearly behind the full method above roughly 80% outliers, the recurrent re-verification idea fails.
Why not just run AnySplat and follow it with a few hundred iterations of 3R-GS-style optimization? That is a reasonable engineering answer, and it is a baseline here at matched wall-clock time. The evidence against it is indirect. 3R-GS needs MCMC Gaussian relocation and an extra geometric loss precisely because short photometric optimization from imperfect poses falls into local minima. Look Gauss, No Pose converges only from rotation errors up to about 15°. A truncated budget is exactly where those minima bite. Our time-matched comparison settles this directly.
Why not just train the feed-forward network with outlier-heavy augmentation and let attention learn to ignore bad matches? If this works, the explicit graph is unnecessary. The no-consistency-layer ablation receives identical augmentation to test it. Our reason for doubt is structural. Repeated-texture outliers are locally consistent and plausible, and only a global agreement test over many matches separates them from the dominant motion. PointMC's 27.4% recall gain on Scan2CAD over multi-model fitting under repetition supports this. Still, it is a hypothesis, and the ablation could refute it.
Proposed Experiments
The main evaluation covers RealEstate10K two-view, DL3DV in two-, six- and eight-view settings, six-view Mip-NeRF 360, and ScanNet. We report PSNR, SSIM, rotation and translation error, and end-to-end latency. Tanks and Temples serves as a cross-domain check against the per-scene methods.
The decisive experiment is a controlled corruption sweep. We inject false matches at ratios from 0% to 99% into the matcher output, and separately evaluate on the DL3DV subsets annotated for reflection and repetitive texture. We plot pose error against outlier ratio for the full method, verify-once, no-consistency, RANSAC-then-feed-forward, PF3plat and AnySplat. The hypothesis predicts that the curves stay together at low ratios and diverge sharply above about 80%, with the full method degrading last.
A second check runs the scale-decoupled consistency module alone on 3DMatch with random per-cloud scaling. Without scaling, it should roughly match PointDSC's reported 93.28% registration recall. With scaling, it should clearly beat rigid PointDSC. This confirms that the decoupling, not the surrounding network, handles scale.
Finally, the per-step readmission analysis described above tests whether the loop actually recovers inliers that the first pass rejected. Gains from the full method without measurable readmission would mean the benefit comes from something other than the claimed mechanism.
References
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting
- AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views
- PointDSC: Robust Point Cloud Registration Using Deep Spatial Consistency
- Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization
- 3R-GS: Best Practice in Optimizing Camera Poses Along with 3DGS
- GS-CPR: Efficient Camera Pose Refinement via 3D Gaussian Splatting
- ReSplat: Learning Recurrent Gaussian Splats
- DeTarNet: Decoupling Translation and Rotation by Siamese Network for Point Cloud Registration
- Efficient Global Point Cloud Registration by Matching Rotation Invariant Features Through Translation Search
- DepthSplat: Connecting Gaussian Splatting and Depth