Calibrating Moving Objects Against the Static Map in Monocular Dynamic SLAM

A single moving camera cannot measure how far away a moving object is. Depth from video comes from watching a point shift as the camera moves, but when the point moves too, its shift mixes camera motion with object motion, and the two cannot be separated. Current monocular dynamic SLAM systems (simultaneous localization and mapping: tracking the camera while building a 3D map) respond in one of two ways. Some throw moving objects away. Others reconstruct them from optical flow, the per-pixel apparent motion between frames, and get scale wrong in ways nobody measures. We propose a third option. The part of the scene that does not move is the only part whose depth is geometrically pinned down across time, so it should serve as the ruler for everything else. In each frame, an off-the-shelf monocular depth network, whose predictions are correct only up to an unknown scale and offset, is calibrated against depth rendered from the static map, using only static pixels. That per-frame calibration is then carried over to the moving pixels, which places people and objects in the map at a scale consistent with the background.

The same calibration does a second job. A monocular SLAM system's scale slowly drifts, and the resulting depth error spreads smoothly across the whole image. Systems that find moving objects by looking for pixels the static model fails to explain, such as De4D-SLAM, therefore get a growing background of false "motion" as drift accumulates. A per-frame, low-dimensional correction absorbs the global part of that error before residuals are read as motion. One calibration step both separates drift from motion and supplies scale for unobservable dynamic depth. Neither job needs ground-truth depth.

Core Claim

In monocular dynamic scenes, dynamic-region depth should be obtained by transfer, not by observation. Specifically, a relative depth prior is fit to the static map on pixels believed to be static, and the fitted mapping is applied to dynamic pixels. Done online inside the SLAM loop, this should do three things. It should reduce depth error on moving objects when evaluated with a single per-sequence scale. It should reduce within-trajectory scale drift. It should make motion segmentation from static-model residuals more stable over long sequences.

The new element is narrow and testable. It is not "add a depth foundation model to dynamic SLAM." It is the rule that the prior's calibration is estimated on static pixels against the persistent map and then transferred to dynamic pixels. It also comes with the claim that a frame-global calibration explains away drift-induced residuals.

Relation to Prior Work

The transfer idea comes from Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimation. That work trains a Depth Scale Alignment module on static objects and applies it to moving ones, reporting a 52.6% reduction in dynamic-region squared relative error on Cityscapes. Three things limit it for our purposes. It is an offline four-network training pipeline. It assumes rigid objects, because it estimates object depth with a rigid-body object network. It produces no map or trajectory. We keep the core move and change its setting. The calibration target becomes the SLAM static map rather than a scene-depth network. The calibration is solved per frame rather than learned, so it covers non-rigid people without any rigidity assumption. The output feeds back into tracking and mapping.

SC-DepthV3 motivates caution about the prior. It distills pseudo-depth from an off-the-shelf single-image model and finds it quantitatively unreliable enough that only depth ordering and surface normals are kept. S3PO-GS reaches a related conclusion for MASt3R pointmaps: it uses them for correspondences and local patch-wise scale alignment, never as raw depth, and only on static scenes. These two papers constrain our design. They are why we fit an explicit low-dimensional mapping per frame, restricted to static pixels, and do not inject raw prior depth. They also predict our main failure mode: errors in the prior that are not affine will not be corrected by an affine fit.

The method sits inside existing Gaussian-splatting SLAM systems, which represent the scene as a cloud of small 3D Gaussian blobs rendered by projecting them onto the image. WildGS-SLAM estimates per-pixel uncertainty from DINOv2 features and down-weights uncertain pixels in DROID-SLAM's dense bundle adjustment and in map optimization. It discards moving objects. De4D-SLAM keeps them in a separate 4D Gaussian stream. It stops gradients from flowing back through static residuals so that the motion model learns only from where the static model fails, and it initializes dynamic Gaussians by lifting optical flow to 3D. That paper itself leaves metric scale and drift open. D2GSLAM does the static-plus-4D decomposition with a depth sensor, flagging motion by comparing observed depth to static-rendered depth. Our proposal can be read as giving a monocular system a substitute for that sensor depth, built from the map rather than from hardware. We do not claim to change the 4D representation itself. We change where dynamic depth and motion residuals come from.

Mechanism

Let $D^{\text{rel}}_t$ be the inverse depth predicted for frame $t$ by a frozen relative-depth network such as Depth Anything. Let $\hat D^{\text{map}}_t$ be the inverse depth rendered from the static Gaussian map at the current pose estimate. Let $w_t(p) \in [0,1]$ be the static probability of pixel $p$ from the system's motion model. Each frame we solve

$$ (a_t, b_t, c_t) = \arg\min_{a,b,c} \sum_p w_t(p)\, \rho\big(a\, D^{\text{rel}}_t(p) + b + c(p) - \hat D^{\text{map}}_t(p)\big) + \lambda \lVert \nabla c \rVert^2 , $$

Here $a_t$ and $b_t$ are a global scale and offset. The term $c(p)$ is a coarse correction field, for example an 8×6 bilinear grid, that absorbs smooth non-affine errors in the prior. The function $\rho$ is a robust loss, and $\lambda$ sets how smooth the correction must be. This is a small weighted least-squares problem and costs well under a millisecond.

The calibrated inverse depth $a_t D^{\text{rel}}_t + b_t + c_t$ is then used in three places. First, on dynamic pixels, where it initializes and supervises the depth of dynamic Gaussians. Flow-lifted 3D velocities are kept, but the flow-derived depth is replaced. Second, on static pixels, where the residual between calibrated prior and map depth replaces the raw photometric residual as input to the motion-probability model. Because the global terms have been fitted away, a slow scale drift no longer looks like motion everywhere. Third, the sequence of fitted $a_t$ is a drift signal. If $a_t$ trends while the camera covers previously mapped ground, the system has detected scale drift and applies a soft constraint pulling newly inserted Gaussians toward the earlier scale.

One circularity is real. The static weights $w_t$ come from residuals that depend on the fit. We break it the way RobustSplat does for transient masks, by bootstrapping. The fit starts at low resolution with a high threshold on $w_t$, and the threshold is relaxed only after it stabilizes. An optional extension adapts the depth network at test time on static pixels, using map depth as pseudo-labels. We treat this as an ablation, not part of the core claim.

Research Program

We implement the calibration module as a drop-in for two hosts. One is WildGS-SLAM, whose map is static-only; there we add a dynamic layer that is placed at calibrated depth but not optimized as 4D Gaussians. The other is a De4D-style static-plus-4D system. Using two hosts tests whether the gain comes from calibration rather than from one architecture. The analysis has three parts. First, we measure how much of the per-frame prior-versus-map discrepancy on static pixels is explained by the global terms versus the correction field versus nothing. This determines whether transfer to dynamic pixels is justified at all. Second, we test whether calibration error measured on held-out static pixels predicts dynamic-region depth error, as the transfer argument requires. Third, we study failure cases where too few static pixels are visible, for example a person filling the frame. Here we expect calibration to fail, and we will report the fraction of static pixels below which it does.

Baselines

The naive reference is the host system unchanged: WildGS-SLAM and the De4D-style host, each with its own depth handling, on identical sequences, frame rates, and keyframe settings. For dynamic depth, the "do nothing" output is the host's own dynamic depth, from flow lifting or from rendering the discarded region.

The strongest tracking competitor on unconstrained dynamic monocular video is WildGS-SLAM. It reports 0.46 cm average ATE RMSE on the Wild-SLAM MoCap split, against 8.71 cm for Splat-SLAM and 47.99 cm for MonoGS. On Bonn, the strongest monocular number in the file is De4D-SLAM, which reports 1.95 cm average ATE. It also reports a 50.4% tracking-error reduction over its DROID-SLAM backbone on TUM, and 0.200 LPIPS on the Wild-SLAM splits. We will rerun both from released code on our hardware with their published settings rather than copy numbers, and report both our rerun and the cited figure. On the MoCap split, at 0.46 cm, there is little room left. Beating WildGS-SLAM there is not the goal; matching it within noise is enough. The claims rest on Bonn, TUM, and the iPhone split, and on dynamic-region depth and drift. No reported result was found on Lacuna for dynamic-region depth error of any monocular SLAM system, so that comparison will be established by our reruns.

The strongest competitor for the depth claim is a pseudo-RGB-D pipeline. A metric-depth foundation model is run per frame, and its depth is fed to D2GSLAM as if it came from a sensor. This is the most direct "simpler" route, and it receives the same depth network and the same compute. For dynamic-depth quality without SLAM, we include SC-DepthV3, which reports 0.142 AbsRel on DDAD and also evaluates on Bonn and TUM, and Manydepth2.

The ablations isolate the key ingredient. In the first, calibration is fit on all pixels rather than on static-weighted pixels. This tests whether static-only fitting matters. In the second, calibration is fit once per sequence rather than per frame. This tests whether per-frame drift absorption matters. In the third, the correction field is removed, leaving only the affine fit. In the fourth, the calibrated prior is used only for dynamic depth and not in the motion residual. Together these separate the two jobs. Beating the hosts means three things: lower dynamic-region AbsRel under a single per-sequence scale, lower within-sequence scale drift, and ATE no worse than the host, all on at least two of the three dynamic benchmarks.

Why not just…?

Why not just feed per-frame metric depth from a foundation model into an RGB-D dynamic SLAM system? This is the real threat, and it is a baseline above, not a strawman. The evidence against it is indirect. SC-DepthV3 found single-image pseudo-depth too inaccurate to use as metric supervision. DVD targets the frame-to-frame scale drift of video depth. Both suggest that per-frame metric predictions disagree across frames, and the static map would absorb that disagreement as geometry. The experiment settles the question directly. If pseudo-RGB-D D2GSLAM matches our dynamic-depth error and drift, the map-anchoring argument fails, and we will report it as such.

Why not just mask people with a segmenter and fit depth scale globally? Semantic masking discards stationary "movable" objects and misses unseen classes, as the masking literature itself notes (DM-SLAM). More to the point, it still says nothing about where the masked objects are. Our all-pixels and per-sequence ablations show what a global fit without static weighting costs. A further ablation replaces our static weights with segmenter masks. If the segmenter masks work as well, the contribution narrows to calibration transfer alone, which is still the core claim.

Why not just rely on the standard similarity alignment used in trajectory evaluation? That alignment fixes one scale for the whole trajectory after the fact. It hides drift rather than correcting it, and it cannot help motion segmentation online. For this reason our depth and drift metrics deliberately allow only one scale per sequence.

Why not use the rigid object network of Mining Supervision? It assumes rigidity, and Bonn and TUM are dominated by walking people. A per-frame calibration has no rigidity assumption. We will report people and rigid objects separately to show where the difference lies.

Proposed Experiments

The main benchmarks are Bonn RGB-D Dynamic, the TUM RGB-D dynamic sequences, and the Wild-SLAM MoCap and iPhone splits. All are run with RGB input only. Sensor depth on Bonn and TUM serves solely as evaluation ground truth, and that is what lets us score dynamic regions. For tracking, we report ATE RMSE. For dynamic-region depth, we report AbsRel and squared relative error inside motion regions, defined from sensor-depth consistency rather than from any method's masks, after a single per-sequence median scale. For drift, we report the spread of window-wise scale factors from similarity alignment over sliding 10-second windows. Rendering quality is reported where the host supports it.

Three checks decide whether the idea holds. First, static-pixel calibration error must correlate with dynamic-region depth error across frames. If it does not, transfer is unjustified. Second, the false-motion rate on static background must grow with trajectory length for the host and not for our variant, a direct test of drift absorption. Third, our system must beat pseudo-RGB-D D2GSLAM on dynamic-region depth and drift at matched compute. If it fails the third check while passing the first two, the defensible conclusion is that calibration helps residual-based motion segmentation but is not the best source of dynamic depth.