A single camera cannot tell a small nearby object from a large distant one. Depth networks trained on millions of images partly resolve this, but they predict depth only up to an unknown stretch and offset, and that stretch drifts from frame to frame. In static scenes the usual fix is to triangulate a sparse set of reliable 3D points from camera motion and bend each frame's depth map to fit them. In scenes with moving people, cars or hands, those reliable points sit on the static background. The depth assigned to the moving object is extrapolated from anchors that may be far away. It is also exposed to the one motion a camera barely sees: an object approaching or receding along the line of sight looks almost exactly like its depth map being rescaled.
We propose to anchor the scale of moving objects with a constraint that does not depend on the background: a rigid part of a moving object keeps its physical size over time. If two tracked points on a forearm or a car door are 30 cm apart in one frame, they should be 30 cm apart in every frame. The method treats these distance-preservation constraints as scale anchors alongside the usual static anchors. Both enter a joint refinement of camera poses and per-frame depth-correction fields, which runs after a sparse tracker has found an approximately correct camera trajectory. The static anchors fix the scale of the world. The size-constancy anchors carry that scale onto moving objects across time, including through stretches where the object is far from any static anchor or moving straight toward the camera.
Core Claim
In dynamic monocular video, the remaining scale error after modern alignment should be concentrated on moving objects and should be worst for motion along the optical axis. Adding per-part size-constancy constraints to a joint pose-and-scale optimization should remove most of that error, while leaving camera tracking accuracy at least unchanged. The claim is narrow on purpose. We do not claim a new tracker, and we do not claim that joint optimization is better than alternating optimization in general. The new ingredient is a scale anchor for dynamic regions, and the experiments are designed to show whether it, and not the optimizer layout, produces the gain.
Where This Sits
The closest system is Back on Track (BA-Track). It is the main foil and supplies most of our front end. A learned 3D point tracker splits observed motion into camera-induced and object-induced parts, which turns dynamic points into "pseudo-static" landmarks for bundle adjustment. Bundle adjustment means jointly refining camera poses and 3D point positions so they reproject consistently into all images. After poses are fixed, BA-Track fits a coarse 2D scale grid per frame that warps the monocular depth map onto the sparse adjusted points. This design leaves two openings. First, scale is solved after pose, so pose errors flow into scale and are never corrected. Second, scale on moving objects is only as good as the depth of the pseudo-static points. Those depths come from the camera-motion component recovered by subtracting predicted object motion, and that subtraction is least reliable precisely when the object moves along the viewing ray. Nothing in BA-Track ties an object's depth in frame $t$ to its depth in frame $t'$ beyond what the motion network supplies.
Locally weighted alignment from Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depth motivates the need for spatially varying correction. It shows that monocular depth errors vary across the image, and that per-pixel scale and shift maps fit to structure-from-motion points improve depth by up to 50%. It also constrains our design. The same paper reports that local fitting degrades when anchors are clustered or absent, and it assumes a static scene. That is exactly the situation of a moving object surrounded by background anchors.
The self-supervised scale-consistency losses of Bian et al. warp depth between frames using estimated ego-motion. That warp is only valid for static points, so it cannot anchor moving objects. Our constraint compares distances within an object, which are invariant to the object's own rigid motion. MoDGS sidesteps inconsistent depth priors on dynamic scenes with an ordinal loss that keeps only near/far ordering. This keeps reconstruction stable but discards metric scale, and it assumes a slowly moving camera. We take the opposite bet and show where metric constraints are trustworthy.
USplat4D shows that monocular uncertainty is much larger along the viewing ray than across the image. It works with fixed camera poses. We reuse that observation to weight anchor residuals differently along and across the ray. Finally, Flow4R predicts per-pixel 3D points and scene flow from image pairs, using the first frame as a persistent scale anchor. It has no multi-frame refinement, which makes it a natural point of comparison for long-horizon 3D track accuracy.
On the tracking side, sparse-then-dense pipelines consistently beat single-strategy trackers (Survey and Evaluation of RGB-D SLAM; DG-SLAM). This is why we keep a sparse tracker as the first stage instead of optimizing everything from scratch.
Research Program
The pipeline has two stages. The first stage is BA-Track's sparse front end, used unchanged: point tracks, motion decomposition, and bundle adjustment produce initial camera poses $T_t$ and sparse 3D points. The second stage is where the contribution lives. For each frame $t$, the monocular network's depth $d_t(u)$ at pixel $u$ is corrected by a coarse per-frame scale grid $S_t(u)$, as in BA-Track, giving the corrected depth $D_t(u) = S_t(u)\,d_t(u)$. A tracked point $i$ observed at pixel $u_i^t$ is lifted to a 3D point $X_i^t$ using $D_t$ and the camera pose $T_t$.
The refinement jointly updates poses and scale grids by minimizing
$$E = E_{\text{reproj}}(T) + \lambda_a \sum_{t}\sum_{k \in \mathcal{A}_t} \rho\big(r_k^{\top}\Sigma_k^{-1} r_k\big) + \lambda_r \sum_{(i,j)\in\mathcal{G}}\sum_{t,t'} w_{ij}\,\rho\big(\lVert X_i^t - X_j^t\rVert - \lVert X_i^{t'} - X_j^{t'}\rVert\big).$$
Each term has a specific role. $E_{\text{reproj}}$ is the dense depth-reprojection residual between consecutive keyframes, restricted to pixels the front end labels static. $\mathcal{A}_t$ is the set of sparse static anchors in frame $t$. Each anchor contributes a residual $r_k$, the difference between the corrected depth-lifted point and the bundle-adjusted point. That residual is weighted by an anisotropic covariance $\Sigma_k$ that is elongated along the viewing ray, following USplat4D's observation, so anchors are trusted less in depth than across the image. $\mathcal{G}$ is a sparse graph of short edges between tracked points on the same moving object, built from points whose motion residuals cluster together. The weight $w_{ij}$ downweights edges whose length varies a lot during an initial pass, which lets articulated or deforming regions opt out. $\rho$ is a robust penalty, and the $\lambda$ values balance the terms.
The third term is the new one. It is unchanged by any rigid motion of the object, so it says nothing about where the object goes. It does say that the object cannot shrink or grow, and that is exactly the quantity the camera cannot observe when the object moves along the optical axis. Absolute scale for the object is still inherited from the static anchors in whichever frames the object's depth is well constrained, such as frames where it touches the ground near background structure. The size-constancy edges carry that scale to every other frame.
The analysis questions are which edges survive the robust weighting on humans, which articulate, versus vehicles, which are rigid, and how far in time a scale anchor can be carried before drift dominates. We will also test whether joint optimization of $T$ and $S$ beats simply alternating BA-Track's two stages. That last question is treated as an empirical one, not a premise.
Baselines
The naive reference point is BA-Track's tracker and bundle adjustment followed by a single global scale and shift per frame, fitted by robust least squares to the sparse adjusted points. This is what most pipelines do when they lift monocular depth into 3D, and it measures how much any spatially varying correction buys.
The strongest existing method is BA-Track itself, with its post-hoc 2D scale grid (Back on Track). It is the only system in the file that tracks a fast-moving monocular camera through dynamic scenes, uses pseudo-static dynamic points, and corrects depth scale per frame. On MPI Sintel it reports an Absolute Translation Error of 0.034, against 0.089 for LEAP-VO and 0.175 for DROID-SLAM. For depth accuracy split by static and dynamic regions, the file contains no reported result for BA-Track, so we will measure it ourselves. To keep the comparison fair, every variant uses the same released tracker weights, the same monocular depth network, the same keyframes and the same scale-grid resolution, and receives the same tuning budget on a held-out split of Sintel. Beating BA-Track means a statistically clear reduction in depth error on dynamic pixels and in temporal scale drift of moving objects, with trajectory error no worse than 0.034 on Sintel.
Two further baselines isolate the ingredients. "BA-Track + local regression" replaces the scale grid with the locally weighted fit of Towards 3D Scene Reconstruction using the same anchors. "Alternating BA-Track" repeats pose adjustment and scale fitting until convergence, which separates the benefit of coupling from the benefit of new constraints. Our ablations remove the size-constancy term, remove the anisotropic weighting, and replace the joint solver with alternation while keeping all terms. The size-constancy ablation is the decisive one.
For 3D tracking of moving points, we compare against Flow4R, which reports an APD3D of 71.1 on PointOdyssey, against 38.5 for SpaTracker and 33.5 for MonST3R. For camera tracking on real dynamic video, we compare against WildGS-SLAM, which reports 0.46 cm average trajectory error on the Wild-SLAM MoCap sequences. We do not expect to beat WildGS-SLAM on trajectory accuracy. That comparison is a check that adding object constraints does not harm camera tracking.
Why not just…?
Why not just fit one global scale and shift per frame? The locally weighted alignment results (Towards 3D Scene Reconstruction) show that monocular depth errors vary across the image, and that this variation costs up to 50% in depth accuracy, so a single scalar is already insufficient for static scenes. More to the point, a global fit to mostly-background anchors gives the moving object whatever scale the background demands, which is the failure we target. The naive baseline is included to measure this directly on dynamic pixels.
Why not just alternate BA-Track's stages until they agree? This is the most serious threat to the "joint optimization" framing, and we concede it may work as well. That is why the claim rests on the size-constancy term, not on coupling. If alternation with all three terms matches the joint solver, we will report alternation as the recommended solver. The prediction that must hold is different: alternation without size constancy should still show scale drift on objects moving along the optical axis, because no amount of re-fitting to background anchors adds information about the object.
Why not just use a scale-consistency warping loss or a stronger video depth model? The geometry-consistency loss of Bian et al. warps depth by camera motion alone. It is therefore biased on independently moving objects, which is the open problem the file notes for that line of work. MoDGS observes that even strong single-view depth models fluctuate on static walls across frames (MoDGS). A stronger depth network shrinks errors within a frame but does not supply information about how scale changes between frames. We nonetheless swap in the strongest available depth network as a sensitivity check. If size constancy stops helping once depth improves, the contribution is small and we will say so.
Why not just mask moving objects, or keep only depth ordering on them? Masking, as in DG-SLAM and WildGS-SLAM, protects camera tracking but throws away the very geometry we want to measure. The ordinal loss of MoDGS keeps moving geometry but abandons metric scale. Both remain useful for comparison. We will report an ordinal-only variant on dynamic regions to show whether metric constraints help or hurt where they are applied.
Proposed Experiments
The main benchmarks are MPI Sintel and PointOdyssey, which provide ground-truth depth and 3D trajectories for moving content. To these we add Bonn RGB-D Dynamic and the dynamic TUM RGB-D sequences, run on RGB only with the sensor depth held out as ground truth, and the Wild-SLAM and Epic Fields sequences for camera tracking. Metrics are absolute relative depth error computed separately on static and dynamic pixels; a temporal scale-drift score, defined as the variance over time of the ratio between estimated and true distances along each moving object's tracked edges; APD3D on PointOdyssey; and trajectory error.
The decisive check slices dynamic-pixel error by the object's motion direction relative to the camera. If the idea is right, the gain over BA-Track and over alternation without size constancy should be largest for approaching or receding objects and near zero for sideways motion, because sideways motion is already well observed. A second check measures gains separately on rigid content (Sintel vehicles and props) and articulated content (humans). It tests whether the robust edge weighting keeps the method from hurting non-rigid regions, and it maps where the assumption breaks. A final check records error as a function of time since the object was last near static anchors, which shows how far size constancy can carry scale before drift accumulates. If none of these slices show a gain beyond what the alternating baseline provides, the hypothesis is wrong and the note's claim fails.
References
- Back on Track: Bundle Adjustment for Dynamic Scene Reconstruction
- Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depth
- Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video
- MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors
- Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction
- Flow4R: Unifying 4D Reconstruction and Tracking with Scene Flow
- Survey and Evaluation of RGB-D SLAM
- DG-SLAM: Robust Dynamic Gaussian Splatting SLAM with Hybrid Pose Optimization
- WildGS-SLAM: Monocular Gaussian Splatting SLAM in Dynamic Environments