Networks that estimate depth from a single image cannot know the absolute size of the world. A toy car up close and a real car far away project to the same pixels. So these networks predict depth only up to an unknown scale, and that scale changes from frame to frame. To turn a video of such predictions into one consistent 3D reconstruction, the usual step is to warp each frame's depth into its neighbour using the camera motion and then solve for the scale that makes them agree. That step breaks in real scenes, because three quite different things all make the warped depth disagree with the predicted depth. The first is scale drift, which is the thing we actually want to correct. The second is occlusion: a surface visible in one frame is hidden behind something in the next. The third is independent motion: a pedestrian or a car moved between frames. Current methods either treat all three as a single class of "outliers" to discard, or they call in an external segmentation network to find moving objects.
This note proposes a way to separate the three signals using only geometry, by exploiting a sign asymmetry. When a static surface is occluded, its warped depth always lands behind the surface the target frame actually sees. It can never land in front, because then the target camera would have seen it. So disagreement in the "too close" direction cannot come from occlusion; it must come from motion or from a scale error. Scale error, in turn, is spatially smooth, while motion is compact and can have either sign. We turn this into a small robust decomposition of the log-depth disagreement map into three parts: a smooth scale field, a nonnegative sparse occlusion part, and a sparse two-sided motion part. The decomposition runs at test time on the output of any monocular depth model. It needs no segmentation and no training.
Core Claim
Let $r(p) = \log \hat D_{s\to t}(p) - \log \hat D_t(p)$ be the log-ratio at pixel $p$ between the source depth warped into the target view and the target's own prediction. We model it as
$$ r(p) = \sigma(p) + o(p) + m(p), \qquad o(p) \ge 0, $$
where $\sigma$ is a smooth log-scale field on a coarse spline grid (inter-frame scale misalignment), $o$ is a sparse nonnegative occlusion term, and $m$ is a sparse, spatially grouped motion term that is unconstrained in sign. We fit this by minimizing
$$ \sum_p \rho\big(r(p) - \sigma(p) - o(p) - m(p)\big) + \lambda_o \|o\|_1 + \lambda_m \|m\|_{2,1} + \lambda_\sigma \|\nabla \sigma\|^2 , $$
subject to $o \ge 0$. Here $\rho$ is a robust penalty, $\|m\|_{2,1}$ is a group-sparsity norm over superpixels, so motion comes in coherent blobs, and the $\lambda$'s are weights.
Two extra geometric facts make the split identifiable rather than just regularized. The first concerns forward and backward warps. A true occlusion in the forward warp has a matching disocclusion, meaning a target pixel with no source correspondence, on the same side of a depth edge in the backward warp. A surface moving away from the camera, which also produces a positive residual, has no such counterpart. The second is that occlusion support must lie next to a discontinuity in $\hat D_t$.
Camera pose is re-estimated only on pixels where both $o$ and $m$ are zero, alternating with the decomposition. Our claim is that this gives substantially lower temporal scale drift and sharper depth at moving-object boundaries than robust scale fitting that lumps the three terms together. The gain should be largest in exactly the regime where outliers are spatially structured rather than scattered: crowded dynamic scenes with many occlusion edges.
Relation to Prior Work
Several papers motivate the gap. Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video introduced exactly the residual we start from, a geometry-consistency loss between warped and predicted depth. It uses the magnitude of that residual as a "self-discovered mask" that down-weights occlusions and moving objects together. We keep the residual but read its sign and spatial structure, so the three causes are separated rather than jointly discarded. The research file also notes that monocular occlusion checks produce false positives when the relative scale between frames shifts. That is the circular dependency we break: the scale field is estimated jointly with the occlusion term instead of before it.
Other work supplies the method. Depth From Videos in the Wild uses a differentiable Z-buffer that keeps the nearer of two points landing on the same pixel. Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry derives occlusion analytically from forward and backward reprojection. Our nonnegativity constraint is the Z-buffer ordering rule turned into a constraint on a decomposition. The difference from both is that we do not assume the scale is already right when applying the rule. We also do not follow the second paper's practice of filling occluded regions with rigid flow, which its authors acknowledge fails when a moving object crosses an occluded area. Our motion term is what flags that case.
A further group constrains the design. Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depth showed that scale error in monocular depth models varies across the image, which is why $\sigma$ is a field and not a scalar. The same work also showed that locally weighted regression degrades when anchors cluster. We therefore keep $\sigma$ on a coarse grid with a smoothness prior, so that masking out large dynamic regions cannot let it overfit. NoPe-NeRF optimizes a per-frame linear scale and shift for monocular depth. It is the global-affine special case of $\sigma$ and assumes a static scene.
Two lines of work come closest in spirit and take the learned route. Back on Track (BA-Track) fits a per-frame 2D scale grid to sparse bundle-adjustment landmarks. It obtains those landmarks from a learned 3D tracker that subtracts object motion, and it reports that tracking under long occlusion is its bottleneck. Flow4R predicts scene flow and a static-pixel weight with a transformer and handles visibility through learned confidence. Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth separates motion from occlusion explicitly, but uses a pre-trained instance segmenter to decide what moves, and does not align scale across the sequence. Our proposal differs from all three in being dense and analytic. It separates the three causes without a segmenter, tracker, or trained visibility head, and the separation is what the scale estimate depends on.
Research Program
The system takes affine-invariant per-frame depth from an off-the-shelf monocular model, plus known intrinsics. It is initialized with pose from a standard two-frame solver. It then alternates three steps for each adjacent pair: the decomposition above, pose re-estimation on the pixels classified as static and visible, and re-warping. Pairwise scale fields are then chained. For long sequences, they are made consistent over a sliding window by a small pose-and-scale graph whose edges are weighted by the fraction of pixels classified as static and visible. This addresses the dead-reckoning drift that Bian et al. acknowledge, although without loop closure it only slows the drift rather than removing it.
Absolute metric scale is out of scope for the core claim, since the decomposition fixes relative scale only. As a secondary study, we will show that a handful of sparse metric anchors, in the style of LiDARTouch, become more effective once they are placed only on pixels classified as static. The reason is that an anchor on a moving car is wrong in exactly the way a mis-scaled frame is wrong.
We will analyze three questions. The first is when the decomposition is identifiable: we will characterize the failure case of a large object moving straight away from the camera in front of a textureless background, where positive residuals lack a disocclusion partner. The second is how sensitive it is to pose error early in the alternation, which is the bootstrapping instability the file documents. The third is how the smoothness weight $\lambda_\sigma$ trades bias for variance in anchor-sparse regions.
Baselines
The naive reference point fits a single least-squares scale per frame to the previous frame over all overlapping pixels. This is what someone would write first, and it shows the raw damage done by occlusion and motion.
The robust version of the same idea is the most honest competitor. It uses the self-discovered-mask weighting from Bian et al., followed by a RANSAC or Huber fit of the same coarse spline scale field we use. Pose is re-estimated on inliers exactly as in our loop. This baseline has identical capacity for $\sigma$, identical pose machinery, identical input depth, and a tuning budget matched by grid search on the same validation split. It differs only in treating the residual as one undifferentiated outlier class.
A segmentation-based variant uses the same robust fit but masks pixels with a pre-trained instance segmenter, following the strategy of DynamicDepth and Depth From Videos in the Wild.
The strongest existing systems are BA-Track and Flow4R, run with their released settings on the same sequences and the same intrinsics. The research file does not include their numbers: no reported result was found on Lacuna for BA-Track, Flow4R, or DynamicDepth on the benchmarks below, so all comparisons will be re-run rather than copied.
Two ablations isolate the key ingredient. One drops the nonnegativity constraint, so that occlusion and motion collapse into a single sparse term. The other drops the forward-backward disocclusion pairing. Beating the robust baseline means lower drift and boundary error at matched runtime on the dynamic subsets. Beating BA-Track or Flow4R on drift in scenes with long occlusions would be a stronger result. We expect to be competitive rather than dominant against them on largely static footage.
Why not just…?
Why not just use robust regression and let RANSAC reject the outliers? This is the real threat. If moving objects and occlusions covered a small, scattered fraction of pixels, a robust fit would recover the scale almost as well as we do. Our argument is that they are neither small nor scattered. They cluster on one side of depth edges and on large foreground objects. The research file notes that locally weighted scale regression is biased when valid support is spatially lopsided, so rejected regions leave the local scale field under-constrained exactly where depth edges are. The robust baseline above has identical capacity. If the gap to it on the high-dynamics subset is within noise, we would conclude that the sign asymmetry buys nothing for scale, and we would reposition the work as an occlusion-and-motion labeller only.
Why not just mask moving things with a segmentation network? A segmenter marks object categories, not motion. It masks parked cars that are useful static anchors, misses non-category movers such as doors, carts, or swaying branches, and says nothing about occlusion. DynamicDepth's reliance on such a segmenter is one of the limitations the research file records. We will report results separately for movers inside and outside the segmenter's label set. We will also report an oracle combination, segmentation plus our occlusion term, to check whether our motion term adds anything beyond a good segmenter.
Why not just use a feed-forward 4D model with learned confidence, such as Flow4R? Learned confidence absorbs all three causes into a single weight. It is also limited by scarce non-rigid scene-flow training data, and Flow4R's alignment is essentially two-view with a first-frame anchor. It may well win on in-distribution driving footage. Our bet is on long occlusions and unusual motion, where BA-Track reports its own tracker failing. We will stratify the evaluation by occlusion duration to test this directly rather than assume it.
Proposed Experiments
We will evaluate on driving video with LiDAR ground truth: KITTI-style raw sequences, and Cityscapes sequences for dense dynamic content. We will also use handheld and legged-robot sequences from FusionPortableV2 to stress non-vehicle motion and varying platforms. Each benchmark is split into a mostly-static subset and a high-dynamics subset, using the fraction of ground-truth points with nonzero scene flow where available and LiDAR-versus-ego-motion inconsistency otherwise.
Metrics are chosen to expose drift rather than hide it. Depth error, as absolute relative error and the δ<1.25 accuracy, is computed after a single scale per sequence rather than the usual per-frame median scaling, which silently forgives drift. Drift itself is reported as the standard deviation of per-frame optimal scale ratios against ground truth. Boundary sharpness is measured as depth error in bands around ground-truth depth edges. Occlusion and motion labelling quality is scored against masks derived from LiDAR where ground-truth flow permits.
The decisive check is a paired comparison of our method against the matched robust baseline on the high-dynamics subset. Sequence-level drift must drop, and boundary-band error must drop, while results on the static subset stay unchanged; an improvement there would suggest the gain comes from something other than the separation. A second decisive check comes from the no-sign ablation. If the error in scene regions containing motion toward the camera rises when the constraint is removed, that is direct evidence that the sign asymmetry is doing the work.
References
- Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video
- Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras
- Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry
- Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depth
- NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior
- Back on Track: Bundle Adjustment for Dynamic Scene Reconstruction
- Flow4R: Unifying 4D Reconstruction and Tracking with Scene Flow
- Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth
- LiDARTouch: Monocular metric depth estimation with a few-beam LiDAR
- FusionPortableV2: A unified multi-sensor dataset for generalized SLAM across diverse platforms and scalable environments