A new class of SLAM systems (simultaneous localization and mapping: estimating a camera's path while building a 3D map) splits the work in two. A pretrained "feed-forward" 3D model reconstructs short windows of video in a single network pass, with no per-scene optimization. A lightweight back-end then stitches these windows, called submaps, into one map. When each submap is distorted only by a global rotation, translation and scale, the stitching is easy. Monocular video, though, has no reliable depth: a single camera cannot tell a small near object from a large far one. So feed-forward models produce submaps that are warped in more complicated ways. A corridor bends slightly, a floor tilts, the far end of a room is stretched. Rigid stitching cannot remove these warps, and they accumulate into drift and doubled surfaces.
We propose to measure how these warps are structured and, if the measurement justifies it, to correct them with a small learned network that runs in the mapping thread. The idea is simple. Consecutive submaps share a few frames. Where they overlap, the disagreement between their depth predictions shows directly how the new submap is bent. A network reads those disagreements together with image features. It then predicts a dense, per-pixel depth correction for every frame of the submap, including the frames no other submap sees. Training is self-supervised: corrected submaps must agree geometrically with each other across views. The corrected keyframe geometry is then sent back to the fast tracking thread as fresh anchors.
Core Claim
The claim is that intra-submap distortion from feed-forward monocular reconstruction is structured but not low-dimensional. It is predictable from image content and from the residuals on the overlapping frames. If so, a learned correction can fix the non-overlapping parts of a submap that any overlap-only alignment leaves untouched. It can do this inside an asynchronous mapping thread, without slowing tracking. Concretely, for a submap with predicted depth $D_i(u)$ at pixel $u$ of frame $i$, the network outputs a log-depth offset $\Delta_i(u)$, and the corrected depth is
$$D'_i(u) = s\, D_i(u)\, \exp\big(\Delta_i(u)\big),$$
where $s$ is the global scale already estimated by rigid alignment. The network is trained so that corrected 3D points from two submaps that observe the same surface coincide.
The claim is falsifiable at the first step. If a per-frame scale and shift, or a smooth low-order warp fitted on the overlap, explains nearly all the distortion, then the learned component is unnecessary. In that case the right contribution is the diagnostic study plus the cheap fix, and we will report it that way.
Relation to Prior Work
The closest system is VGGT-SLAM 2.0. It chains submaps from a pretrained VGGT model through a keyframe factor graph in which overlapping frames share pose and intrinsics, so only a one-degree-of-freedom scale is optimized. Its stated limitation motivates this work: the projective geometry inside a submap is treated as rigid, so non-linear depth distortion cannot be corrected. We keep its factor graph and add a non-rigid stage on top.
Two other systems attack monocular drift from the tracking side. Flash-Mono uses a recurrent transformer whose hidden state carries temporal context, and closes loops with Sim(3) constraints. It reports a "forgetting horizon" and has no explicit cross-view geometric losses at inference time. DepthGS lifts single-frame UniDepthV2 depth into Gaussians and lets the optimizable Gaussians absorb depth error. Its map remains bound to the single-frame depth estimator. Both systems correct poses and leave the depth field's shape alone, which is the part our proposal targets.
S3PO-GS is the most important competitor in spirit. It aligns MASt3R geometry to the SLAM map by solving local, patch-wise scale factors, which is already a non-rigid correction. It works only where the map and the prior overlap, and it solves the factors by iterative optimization. Our proposal differs in two ways. It predicts corrections for regions without overlap, and it does so in one forward pass. Whether those two differences matter is exactly what our experiments test.
AMB3R shows that a 3D sparse-voxel backend enforcing "one physical point, one location" sharply improves feed-forward geometry. It reduces TUM visual-odometry error from 7.1 cm to 3.2 cm ATE RMSE. It operates on chunks, not on a continuous stream of submaps, and its Point Transformer backend is heavy. We take from it the lesson that explicit 3D consistency reasoning helps. We apply that lesson across submap boundaries with a much smaller model.
For the training signal we reuse methods from multi-view geometric supervision. These are the symmetric 3D-3D consistency loss of PF3plat and the overlap-window depth-warping consistency of LCP-Fusion. We weight residuals by predicted uncertainty, following Trust3R. Trust3R shows that the raw confidence scores of foundation models track true error poorly, and that evidential weighting lowers downstream SLAM tracking error. One constraint comes from Trust3R as well: its uncertainty is unimodal and degrades beyond about 10 m. We therefore restrict the main claims to indoor scenes and treat outdoor driving data as a stress test.
Research Program
The work has three stages, and each one gates the next.
The first stage is a diagnostic study of distortion structure. We run a frozen VGGT in the VGGT-SLAM 2.0 submap configuration on ScanNet, TUM RGB-D and Replica. After the best rigid Sim(3) alignment of each submap to ground-truth geometry, we fit progressively richer correction models to the residual depth error. In order, these are a per-frame scale, a per-frame scale and shift, a smooth low-order spline per frame, and an oracle per-pixel correction. For each model we record how much error remains.
We also ask the question the learned approach depends on. Do the residuals measured on the overlapping frames predict the error on the non-overlapping frames? We estimate this by regressing one on the other, with and without image features. If simple models close most of the gap, or if overlap residuals carry no information about the rest of the submap, we stop and report that.
The second stage builds the correction network. Its inputs are frozen VGGT features for the submap's frames, the overlap residual maps, and per-pixel uncertainty. Its output is the offset field $\Delta$. The architecture is a small attention model over frame tokens that carries the residual signal from overlapping frames to the other frames. It is much lighter than AMB3R's backend.
Training uses long sequences from ScanNet and DL3DV-10K. The loss combines three terms. The first is a symmetric 3D-3D consistency term between corrected submaps on shared frames. The second is a depth-reprojection term on frames that are held out from the overlap: these frames are seen by a later submap, but that later submap's information is withheld from the network at input time. The third is a smoothness prior. Ground-truth depth is used only for evaluation, so the method stays self-supervised.
The third stage integrates the network into the decoupled SLAM loop. The correction runs in the mapping thread whenever a submap is added. The tracking thread keeps running on the newest frames. Corrected keyframe pointmaps are pushed back to the tracker through a queue, which addresses the problem of the tracker working from stale anchors. We measure the latency budget explicitly, because the correction must complete before the next submap arrives.
Baselines
The naive reference point is VGGT submaps chained with rigid Sim(3) alignment only and no factor-graph refinement. This shows how much distortion there is to fix. The direct parent is VGGT-SLAM 2.0 itself. It is the strongest feed-forward submap SLAM system in the file and isolates the effect of adding non-rigid correction on top of its rigid factor graph. The file reports only a relative figure for it: about 23% less pose error on TUM RGB-D than the original VGGT-SLAM, and 8.4 FPS on an RTX 3090. No absolute ATE value for VGGT-SLAM 2.0 was found on Lacuna, so we will rerun it.
The strongest reported tracking number on TUM in the file is AMB3R at 3.2 cm ATE RMSE (versus 7.1 cm for VGGT) at 4.2 FPS on an RTX 4090. We treat it as the accuracy bar for real indoor data. Beating it means matching or improving ATE while sustaining higher throughput in a fully online setting.
On ScanNet, the file reports 7.22 cm ATE RMSE for LCP-Fusion. That is an optimization-based RGB-D method, so we include it as context rather than as a like-for-like competitor. Flash-Mono is the strongest feed-forward monocular system on ScanNet. It reports the lowest ATE on almost all ScanNetV1 sequences at about 12 FPS, but no per-sequence values were found on Lacuna. We will run its released configuration if one is available and otherwise note its absence. On Replica, DepthGS reports 15.49 cm average ATE RMSE, which is the monocular reference there.
The ablation that isolates the key ingredient is an overlap-only optimizer. It uses the same backbone, submaps and uncertainty weights. It fits per-patch scale factors on the overlap by least squares, as S3PO-GS does, and propagates them to non-overlapping frames by nearest-neighbor or smooth interpolation, with no learned propagation.
Further ablations remove the uncertainty weighting, remove the feedback to the tracker, and replace the learned field with the best low-order spline from stage one.
All methods share the frozen VGGT backbone, the same submap length and overlap, the same keyframe policy, and the same GPU. Learned components receive equal training data and an equal hyperparameter sweep. Optimizers receive an iteration budget matched to the network's wall-clock latency. We report ATE RMSE, depth AbsRel on non-overlapping frames against ground truth, mesh accuracy and completeness on Replica and ScanNet, and end-to-end FPS. Winning requires a clear improvement over the overlap-only optimizer on non-overlap depth error and on ATE, at equal latency.
Why not just…?
Why not just fit per-patch scale factors on the overlap, as S3PO-GS does? This is the real threat. If stage one shows that distortion is smooth and well captured by a few scale parameters per frame, this alternative will match us and should be adopted instead. Its known weakness is that it has no signal where submaps do not overlap. With short overlaps, most of each submap sits in that blind region. The overlap-only ablation, evaluated on depth error in non-overlapping frames, settles the question directly. We commit to reporting it even if it wins.
Why not just run a heavier 3D backend like AMB3R over larger windows? Larger windows reduce the number of seams but do not remove them. AMB3R's own limitations note that it is chunk-based, lacks online tracking and loop closure, and runs a costly Point Transformer. We will run AMB3R with windows as large as fit in memory and compare accuracy against throughput. If it dominates at real-time rates, the learned correction is not needed.
Why not just use a single batch foundation model over the whole sequence? Batch models avoid seams entirely but cannot run online. AnySplat reports about 33 seconds for 200 views, and long sequences hit memory ceilings. MapAnything shows that metric factorization helps scale but remains a batch method. We use offline batch reconstruction of short sequences as an accuracy ceiling, not as a deployable competitor.
Why not just make the global factor graph richer, for example with per-keyframe scale variables? A per-keyframe scale is one of the stage-one correction models, so the diagnostic study covers this alternative directly. If per-frame scale suffices, the factor graph should absorb it and we will say so.
Decisive Checks
Three outcomes decide the project. The first is the stage-one fraction of residual depth error left after per-frame scale and shift. If that fraction is small, the learned approach is unnecessary. The second is whether the learned field beats the overlap-only optimizer on non-overlap depth error and ATE, at matched latency, on ScanNet and TUM. We will specifically include long corridor and multi-room sequences, where bending accumulates most. The third is whether feeding corrected anchors back to the tracker lowers ATE beyond what correcting the map alone achieves. If it does, the decoupled design is doing real work rather than merely hiding latency. KITTI and Waymo serve as out-of-distribution stress tests, where we expect uncertainty calibration to limit the gains.
References
- VGGT-SLAM 2.0: Real-time Dense Feed-forward Scene Reconstruction
- Flash-Mono: Feed-Forward Accelerated Gaussian Splatting Monocular SLAM
- Pseudo Depth Meets Gaussian: A Feed-forward RGB SLAM Baseline
- Outdoor Monocular SLAM with Global Scale-Consistent 3D Gaussian Pointmaps
- AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backend
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting
- LCP-Fusion: A Neural Implicit SLAM with Enhanced Local Constraints and Computable Prior
- Trust It or Not: Evidential Uncertainty for Feed-Forward 3D Reconstruction with Trust3R
- AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views
- MapAnything: Universal Feed-Forward Metric 3D Reconstruction