Telling Scale Drift from Stretching in Monocular Deformable Mapping

A single moving camera cannot measure absolute distance. Monocular mapping systems therefore start with an arbitrary scale, and that scale slowly drifts as the camera travels. The classical fix is to let each camera pose carry its own scale factor. This means optimizing over similarity transforms (rotation, translation and one scalar scale, the group $Sim(3)$) rather than rigid transforms. When the camera revisits a place, the mismatch is then spread back along the trajectory. That fix assumes the world itself is rigid. When the scene contains something that genuinely stretches or shrinks, such as a cloth being pulled, a breathing body, or tissue under an endoscope, the camera sees the same symptom it would see from drift: things look bigger or smaller than before. A system that allows both scale drift and surface deformation as free variables has no principled way to decide which one happened. Today the only working answer is to bolt on an inertial sensor.

We propose a visual-only pose graph that splits the two explicitly. Every apparent change in size is divided into a single per-keyframe trajectory scale, which loop closures can correct, and a spatially varying surface stretch field whose average is forced to zero. Under this split, uniform dilation is by construction attributed to drift, and non-uniform dilation is attributed to deformation. The split is switched off whenever the image evidence cannot support it. The practical target is twofold: monocular deformable odometry without an inertial measurement unit (IMU), and stitching the chunk-wise, arbitrarily scaled 3D outputs of feed-forward reconstruction models that generative world models now rely on.

Core Claim

The ambiguity between trajectory scale drift and surface deformation is a gauge freedom, meaning a direction in parameter space that the data cannot see. It is best removed by an explicit gauge choice rather than by soft regularization. Once that choice is made, ordinary $Sim(3)$ loop closure becomes usable in deforming scenes again. We claim that this split, combined with a visual-only test for when deformation can be observed at all, recovers most of the trajectory accuracy that inertial anchoring provides on deformable benchmarks. We further claim that it keeps reconstructed geometry from absorbing drift as fake stretching, or stretching as fake drift.

Where the Directions Meet and What Is Missing

Scale-aware pose graph optimization supplies the backbone. Double window optimisation for constant time visual SLAM showed that putting scale into keyframe poses lets a loop closure redistribute drift over the whole graph. RegGS applies the same idea to register independently predicted local Gaussian models in $Sim(3)$, parameterizing scale as $\log s$. Both assume one global similarity transform explains every difference between views, so both are the method we reuse and also the assumption we relax.

Non-rigid modeling supplies the deformation side. DefVINS is the closest existing system. It splits state into a rigid camera part anchored by an IMU and a deformation graph built from feature tracks, and it keeps the deformation variables dormant when an online condition number says ego-motion and deformation cannot be separated. It motivates our work and constrains it, because it shows the separation problem is real and that gating matters. Its scale comes from the accelerometer, however, and it has no loop-closing similarity graph. Con-NRSfM shows that a local conformal factor (angle-preserving stretch) can be recovered separately from depth using differential geometry. We borrow its stretch parameterization, but Con-NRSfM solves local surface geometry without any trajectory-level graph. Non-rigid Reconstruction with a Single Moving RGB-D Camera anchors the camera on a segmented rigid background and uses a coarse-to-fine deformation graph. It relies on metric depth from the sensor, which is exactly what we do not have.

Geometric inconsistency in generative world models is the application regime. FR3D predicts future 3D scenes by distilling from a frozen point-map model. Its authors report cumulative scale drift over long rollouts and have no back-end to correct it. FantasyWorld obtains strong 3D consistency only in quasi-static scenes and lacks loop closure. VerseCrafter represents moving objects as single Gaussian ellipsoids and leaves background scale uncorrected. These systems produce exactly the input our back-end consumes: short chunks of 3D points with unknown, inconsistent scale that sometimes contain deforming content. We do not modify their generators. We treat their geometric output as noisy measurements to be reconciled.

Research Program

The state consists of one $Sim(3)$ pose $T_i$ per keyframe $i$, with scale $s_i$, plus an embedded deformation graph. The deformation graph is a sparse set of nodes $k$ on the tracked surface. Each node carries a local rotation, a translation and a log conformal stretch $\phi_k(t)$. The central constraint, the gauge, is that the stretch field has zero weighted mean at every keyframe:

$$\sum_k w_k\,\phi_k(t_i) = 0 \quad \text{for every keyframe } i,$$

Here $w_k$ is the node's surface area share. This forces any uniform size change into $\log s_i$, where loop closures can see it and correct it. Only spatially uneven stretch remains as deformation.

The energy minimized is:

$$E = \sum_{(i,j)} \| \log(T_{ij}^{-1} T_i^{-1} T_j) \|^2_{\Sigma_{ij}} + \sum_{\text{obs}} \| \pi(T_i, \mathcal{W}_{t_i}(X)) - u \|^2 + \lambda \sum_{k \sim l} \big( \|\phi_k - \phi_l\|^2 + \text{ARAP}_{kl} \big).$$

The terms are defined as follows. The first term sums over odometry and loop-closure edges with measured relative similarity $T_{ij}$ and covariance $\Sigma_{ij}$. The second term compares projected points to their image observations $u$; there, $\mathcal{W}_{t}$ warps canonical points $X$ by the deformation graph and $\pi$ projects into keyframe $i$. The third term regularizes neighboring nodes $k \sim l$ toward smooth stretch and as-rigid-as-possible rotation, weighted by $\lambda$.

Loop closures are detected on appearance and verified with a robust $Sim(3)$ fit restricted to points the deformation model labels as low-stretch.

Following DefVINS but without inertial data, we compute a visual observability score. This is the smallest singular value of the Hessian block coupling $\{\log s_i\}$ to $\{\phi_k\}$, after the gauge has been applied. While the score is low, the stretch field is frozen at its last value and the node is treated as rigid with only the global scale free. This test is the second ingredient and is ablated separately.

The same back-end runs in a second mode for feed-forward reconstruction. Each chunk of point maps from a CUT3R-style model, the teacher FR3D distills from, becomes a node with its own $Sim(3)$ pose. Overlapping chunks provide edges. Optionally, a RegGS-style optimal-transport registration provides correspondences, extended with the per-node stretch field so that deforming people do not pull the chunk registration apart.

Baselines

The naive reference is monocular rigid tracking with no deformation model: ORB-SLAM3 in monocular mode with its own $Sim(3)$ loop closure, run on frames identical to those our method receives. In the chunk-stitching mode, the naive reference is simply concatenating feed-forward chunks after aligning each consecutive pair by median depth ratio. Our method should beat it clearly on high-deformation sequences. If it does not, the deformation side contributes nothing.

The strongest visual-only competitor is NR-SLAM, which DefVINS reports outperforming. It models deformation but has no explicit scale-versus-stretch gauge. It will be run with its released settings and the same feature front-end where possible, and given a tuning budget equal to ours on a held-out sequence.

The strongest overall system is DefVINS itself, which has access to an IMU. On VIMandala sequence R6 it reports an 80% reduction in Absolute Trajectory Error relative to ORB-SLAM3. On the "Extreme" level of the Drunkard's Dataset it reports roughly a 40–45% reduction. Both figures come from DefVINS. No absolute per-sequence ATE values were found on Lacuna. We therefore run ORB-SLAM3 ourselves to recover a common reference and report relative reductions on the same scale. DefVINS is an upper bound, not a target. A visual-only result that recovers at least half of its relative gain over ORB-SLAM3 on Drunkard's Extreme would count as success, and matching it would be surprising.

For the stitching mode, the strongest comparison is RegGS-style rigid $Sim(3)$ optimal-transport registration on the same chunks. No reported result was found on Lacuna for RegGS on dynamic data. Beating it means lower per-segment scale error and lower Chamfer distance on dynamic sequences, while staying within noise on static RE10K and ACID clips.

Three ablations isolate the key ingredient. The first replaces the hard zero-mean gauge with a soft penalty on the mean stretch, testing whether an explicit gauge beats regularization. The second removes the observability gate. The third removes loop closures entirely, showing how much of the gain comes from global correction rather than local modeling.

Why not just…?

Why not just mask out the deforming regions and run $Sim(3)$ SLAM on the static background? In driving-style scenes, where background dominates and movers are people or cars, this is a real threat, and we expect it to match us on camera trajectory accuracy there. We will report that honestly. It fails in two regimes the file documents. The first is when the deforming surface fills the view, as on VIMandala and Drunkard's, where DefVINS notes there is too little static structure to track. The second is when the geometry of the movers is the goal, which Robust Dense Mapping for Large-Scale Dynamic Environments identifies as the blind spot of pruning. We run a masking baseline with Grounded-SAM2 masks plus ORB-SLAM3 on every benchmark and report the regime where it wins.

Why not just approximate movers with rigid pieces? Piecewise-rigid models need no stretch variable at all, so scale drift would be the only explanation for size change. RigidFlow and Superpoint Gaussian Splatting show such clustering tears on smooth stretching like cloth and skin. The NRSfM benchmark's Stretching and Deflation modes directly test this. We include a supervoxel variant of our graph, with $\phi$ fixed at zero, as a baseline. If it matches us on Articulated Motion but loses on Stretching, that is the expected and informative pattern.

Why not just anchor scale with a monocular metric-depth network on every frame? This may be the most dangerous alternative, since it attacks drift directly. No reported result was found on Lacuna for depth-foundation-model scale anchoring on deformable benchmarks, so we test it. We rescale each keyframe by the median ratio between the predicted metric depth and the tracked depth, and also insert those ratios as unary priors on $\log s_i$ in our graph. If per-frame anchoring alone matches our trajectory accuracy, the gauge split is unnecessary for trajectories and its value reduces to correct stretch attribution, which we measure separately. If the priors help inside our graph, we keep them as a complementary signal.

Why not just add an IMU? For robots that have one, DefVINS is the answer, and we treat it as the upper bound. Internet video, historical footage and generated world-model rollouts have no inertial data, which is precisely the regime where FR3D reports unresolved scale drift.

Proposed Experiments

The first study uses Drunkard's at all four deformation levels and VIMandala sequences R0–R6, discarding the IMU stream. We report ATE after a single whole-trajectory $Sim(3)$ alignment; a single alignment exposes drift, whereas per-segment alignment would hide it. We also report the per-segment scale error, the deviation of the locally fitted scale from the global one, and the number of tracked frames.

The decisive check for the gauge claim is a controlled synthetic test. We apply a known uniform dilation to the scene halfway through a loop, then separately a known non-uniform stretch. A correct split should send the first almost entirely into $\log s_i$ before loop closure, or recognize it as physical when the loop closure contradicts drift. It should send the second into $\phi$. The soft-penalty ablation is expected to smear both.

The second study uses the NRSfM animatronic benchmark from A Benchmark and Evaluation of Non-Rigid Structure from Motion. It covers the Line and Zigzag paths, where observability is poor, and the Stretching and Deflation modes, and it compares stretch accuracy against Con-NRSfM. Con-NRSfM reports 0.81% 3D error on synthetic conformal data. We expect our trajectory-level graph to trade some local accuracy for consistency over long paths, and we will report the trade-off rather than claim parity.

The third study stitches CUT3R-style chunk predictions on Dynamic-RE10K and on static RE10K and ACID as a sanity check. It measures cross-chunk scale consistency against ground-truth camera paths, and Chamfer error on the dynamic subjects where ground truth permits. Success means lower long-horizon scale error than rigid $Sim(3)$ stitching on dynamic clips, with no loss on static ones.