Rigid Objects as Their Own Ruler: Fixing Foreground Drift in Video Depth

Today's best video depth models take a strong single-image depth network and add temporal layers so that depth stops flickering from frame to frame. Long videos are processed in overlapping chunks, and the chunks are stitched together by fitting one scale and one offset per chunk to the whole frame. That single fit assumes that all disagreement between chunks is caused by the camera. A car driving toward the camera, or a person walking across a room, changes its depth independently of the background. A whole-frame fit therefore has to trade the two against each other: either the background drifts, or the moving object slowly stretches, shrinks, and swims. We propose to correct moving objects with a geometric fact that holds in any domain. The points on a rigid object keep the same 3D distances from one another over time, so the object's apparent size and its predicted depth must change in lockstep. Any network output that breaks this is wrong, and the broken constraint also says how to fix it.

In practice this is a small post-processing solver that attaches to any video depth model. The background is aligned with the usual per-chunk scale and offset. Each moving rigid object gets its own per-frame scale and offset, chosen so that its back-projected shape stays rigid over the whole clip. Where the object visibly rests on the ground, a contact term ties its placement to the background. The same quantity, the amount by which a clip violates rigidity on its moving objects, also serves as a label-free measure of foreground temporal consistency. Current benchmarks do not report this, because their temporal metrics average over the whole frame.

Core Claim

Monocular depth is known only up to an unknown scale and shift. Current long-video methods resolve that ambiguity once per chunk for the whole frame: Video Depth Anything anchors sliding segments to keyframes, DVD stitches windows with a closed-form global affine least-squares fit, and StableDPT replaces stitching with cross-attention to global keyframes. None of them separates moving objects from the background. Our claim is that the remaining temporal error in dynamic scenes sits mostly on independently moving objects, and that rigidity is enough to remove most of it without retraining, optical flow, or bundle adjustment. Rigidity is a constraint that ignores appearance and holds equally for indoor and outdoor scenes.

Concretely, take a rigid object $o$ and a set of points on it tracked across frames. Let $p_i^t$ be the homogeneous pixel of point $i$ at frame $t$, $d_i^t$ the depth predicted by the base network after background alignment, $K$ the camera intrinsics, and $(a_t, b_t)$ an unknown per-frame scale and shift for this object. The corrected 3D point is $X_i^t = (a_t d_i^t + b_t) K^{-1} p_i^t$. We solve

$$\min_{\{a_t, b_t\}, \{L_{ij}\}} \sum_t \sum_{(i,j)} \left( \lVert X_i^t - X_j^t \rVert - L_{ij} \right)^2 + \lambda \sum_t \left[ (a_{t+1}-a_t)^2 + (b_{t+1}-b_t)^2 \right],$$

where $L_{ij}$ is the unknown but time-constant 3D distance between points $i$ and $j$, and $\lambda$ weights a mild smoothness prior. Rigidity fixes the object's depth trajectory up to one overall scale. We fix that remaining scale in one of two ways. If the object's lowest tracked points lie on a background surface, we add a ground-contact term, the differentiable counterpart of the connectivity prior of Dense Monocular Depth Estimation in Complex Dynamic Scenes. Otherwise we set it to the network's own time-averaged prediction, so the correction redistributes the scale over time without inventing a new one. Each object's problem has two unknowns per frame plus the pair distances, so it is small and runs in milliseconds.

Relation to Prior Work

The papers above that treat the whole frame with one alignment motivate this work, because they leave the gap open. Gap 4 of the source analysis, that global affine coherence conflicts with independent foreground motion, is exactly what we test.

STATIC is the one video depth method that does separate static from dynamic regions. It finds them from differences in surface normals and routes them through separate attention modules. Its reach is limited to two consecutive frames, it is trained with normals derived from ground truth, and it does not constrain the depth trajectory of the moving region over long spans. Our solver works over the whole clip and needs no training.

Back on Track (BA-Track) is the closest system and a real competitor. It tracks 3D points, subtracts the motion caused by objects so that moving points become "pseudo-static" landmarks for bundle adjustment, and then corrects dense monocular depth with a per-frame 2D scale grid fitted to the sparse adjusted points. We differ in three ways. BA-Track requires calibrated intrinsics and a full camera bundle adjustment; we need only approximate intrinsics and no camera solve. Its scale grid is spatially smooth and only scales depth; our per-object scale-and-shift respects object boundaries. And it enforces consistency indirectly through the landmarks, whereas we impose rigidity directly. Whether these differences matter is an empirical question that the experiments are built to answer.

Two other lines constrain the design. Consistent video depth estimation masks people out of its geometric loss, so moving objects are left to the single-image prior; this is the gap we fill. Video Depth Estimation by Fusing Flow-to-Depth Proposals shows that triangulation grounded in geometry transfers across datasets but flags moving objects as low-confidence outliers. From it we take the lesson that geometric constraints generalize, but we apply one, rigidity, that stays valid when objects move.

Our method is built from existing components: an off-the-shelf point tracker, such as the front end BA-Track trains on Kubric, and per-frame instance masks. Tracker failure under long occlusions, which BA-Track reports, is inherited. The solver therefore has to work from short track fragments, and pair distances are shared only between fragments that overlap in time.

Research Program

The first stage is diagnostic, and it could stop the project early. We will run Video Depth Anything, DVD, DepthCrafter, and StableDPT on dynamic clips with ground-truth depth, split every error and temporal-consistency metric into moving-object and background pixels, and compute the rigidity residual on moving rigid objects. If foreground error per pixel is not clearly worse than background error, or does not grow with window length and object approach speed, the premise fails and we report that as a negative finding.

The second stage builds the solver. It has four parts: background alignment with the base method's own stitching, per-object rigidity solves over tracked fragments, the optional contact term, and a fallback that leaves the object's prediction untouched when its rigidity residual is unexplained. That last case covers non-rigid objects, where the residual cannot be driven down. For people we will test a piecewise-rigid variant that runs the solve on separately tracked body parts. We expect gains to be weaker there, and we will report them separately.

The third stage asks whether the benefit can be learned. We will use the rigidity residual as a training loss on the temporal head of a Video Depth Anything-style model, trained on synthetic dynamic video, and compare it with the post-hoc solver.

Baselines

The naive reference point is the base video depth model with its own stitching and no foreground treatment. Our main instance is DVD, because the file reports the lowest zero-shot errors for it: AbsRel 6.7 on KITTI and 5.5 on ScanNet, as reported in DVD. Video Depth Anything and StableDPT serve as second hosts, to show the solver is not tied to one network. Every correction method receives the same base depth maps, the same intrinsics, and the same masks and tracks, and the solver's single weight $\lambda$ is tuned only on a held-out synthetic split. Beating the naive reference means a statistically reliable reduction in moving-object AbsRel and temporal error on Sintel, Bonn, and KITTI dynamic regions, with background metrics unchanged within noise.

The strongest existing corrector is BA-Track's scale-grid refinement applied to the same base depth, run with the authors' tracker and given the ground-truth intrinsics it requires. On Sintel the file reports BA-Track's camera error (ATE 0.034, versus 0.175 for DROID-SLAM, from Back on Track), but no dynamic-region depth number; no reported result found on Lacuna for BA-Track foreground depth error. Beating it means lower moving-object error at equal or lower runtime, and remaining usable when intrinsics are only roughly known, which we test by perturbing focal length by up to 20%.

The ablation that isolates the key ingredient is per-object stitching without rigidity. It uses the same masks and gives each object its own scale and offset fitted only on overlapping window frames, exactly as the global stitching does for the whole frame. This tells us whether rigidity adds anything beyond separating objects from the background. A second ablation turns the contact term off, and a third replaces predicted masks with ground-truth masks to bound how much mask errors cost.

Why not just…?

Why not just fit a separate scale and offset per segmented object when stitching windows? This is the most serious threat, and it is our main ablation. Overlap fitting only aligns one window to the next. It cannot correct drift within a single window, and it inherits whatever scale error the first window has, so errors accumulate over long clips. Rigidity constrains every frame against every other frame on the same object. If per-object stitching matches the full solver on long Sintel and Bonn clips, the rigidity term is not earning its place, and we would reduce the contribution to the diagnostic and the metric.

Why not just use BA-Track's dense scale grid, which already corrects depth using dynamic points? It may well be competitive. The file offers two grounds for doubt: a spatially smooth grid only rescales depth and can bleed across object boundaries, and the method needs calibrated intrinsics and a full camera bundle adjustment. The head-to-head comparison above, including the perturbed-intrinsics condition and the error band near object boundaries, settles this directly. If BA-Track wins on accuracy, our remaining claim is lower cost and robustness to unknown intrinsics, and we will say so.

Why not just train the temporal head longer, or with a flow-warping loss? Video Depth Anything moved away from flow-based losses because flow noise compounds over time. STATIC reports that flow is unreliable in exactly the occluded, dynamic regions at issue, and it outperformed the flow-based MAMo on KITTI (RMSE 1.977). Whole-frame temporal losses also weight foreground pixels only by their area. Our third stage tests the learned alternative directly: the rigidity residual as a loss, against a flow-warping loss, with equal data and compute.

Proposed Experiments

We evaluate zero-shot on MPI Sintel (final pass), which has large rigid and non-rigid motion. On Bonn RGB-D Dynamic and the dynamic sequences of TUM RGB-D we use humans and handled objects. On KITTI we evaluate LiDAR ground-truth depth restricted to car masks, and on Waymo we test transfer to outdoor driving. All metrics are split into moving-object and background regions: AbsRel, threshold accuracy, and temporal consistency (temporal gradient error and warping error computed with ground-truth flow where Sintel provides it). We also report the rigidity residual and wall-clock time per frame.

We expect the gain to grow with clip length and with how fast objects approach or recede from the camera, since those are the conditions under which global stitching must compromise. Results will be plotted against both variables. For Sintel, Bonn, and dynamic-region KITTI depth under these host models, the file contains no split foreground numbers, so all such numbers will be produced by us under one shared protocol.

Three outcomes would be decisive. First, moving-object error should fall substantially while background error stays unchanged. Second, the full solver should beat per-object stitching without rigidity on clips longer than two windows. Third, the rigidity residual should correlate with ground-truth foreground error across methods, which would justify it as a metric that needs no labels. If only the third holds, the metric alone is still worth publishing.