Contact-Anchored Depth for Moving Objects in Self-Supervised Monocular Video

A depth network can be trained from ordinary video without any 3D labels: predict depth for one frame, use it to warp a neighboring frame into that view, and penalize the pixels that fail to match. This works for the static world. It breaks for anything that moves. The trouble is not noise. It is a real ambiguity. If a pedestrian is allowed its own motion, then a pedestrian twice as far away and moving twice as fast produces exactly the same image motion. The warping loss cannot tell these two cases apart, so current methods either throw moving pixels away or fall back on rigid-body and fixed camera-height assumptions. Our proposal is to break the tie with a fact the video already contains. Almost every moving thing in a street scene touches the static world, through feet on the pavement or tires on the road. The depth of the static surface at that point of contact is well supervised by ordinary warping. It therefore fixes the object's distance, and this holds whether or not the object is rigid.

Concretely, we add one training signal to a standard self-supervised depth network. For each region flagged as moving, we find the pixels where its lower boundary meets static background. We then require the predicted depth there to equal the background depth just below. The background depth is held fixed when computing this loss, so the object cannot pull the road toward itself. Everything trains end to end in a single network. The method needs no per-object scale-alignment stage and no rigid motion model. In its main variant, it also needs no external instance segmenter. The claim we want to test is narrow. A local contact constraint should recover metric-consistent depth on moving objects at least as well as learned scale alignment. It should also clearly beat that approach on non-rigid movers such as pedestrians and cyclists, where rigid formulations are known to fail.

Why this gap is real

The ambiguity is documented directly in the prior work, and that work also shows how it is currently handled. Monodepth2 uses a per-pixel minimum over source frames plus "auto-masking": any pixel that warping does not explain better than leaving the frame unwarped is dropped from the loss. This removes the "infinite depth" holes that appear on cars driving alongside the camera. It also leaves moving objects with no geometric supervision at all. Insta-DM models each instance as a rigid body moving in 6 degrees of freedom, using off-the-shelf instance masks and optical flow. It needs a camera-height constraint to avoid the infinite-depth trap, and it cuts dynamic-region error on KITTI from about 0.396 to 0.178 AbsRel. Mining Supervision for Dynamic Regions makes the sharpest observation. A single-frame network cannot see whether an object is moving, so a module that predicts each object's scale can be trained on stationary objects and then applied to moving ones. Its multi-stage pipeline reports a 52.6% reduction in dynamic-region Sq Rel on Cityscapes. Both of these methods assume rigid objects, and the second needs four separately trained networks.

ProDepth takes a different route. It detects moving regions by checking where a multi-frame matching volume disagrees with a single-frame prediction, and it fuses the two as probabilities, so it needs no segmenter. On moving pixels, however, it simply trusts the single-frame cue. Nothing geometric anchors that cue to the background's scale.

The contact idea combines two insights from this work. The first comes from Mining Supervision: depth for moving objects should be borrowed from what the static scene reliably supervises. The second comes from ProDepth: moving regions can be found without external masks. What we replace is the source of the anchor. Prior methods learn a scale for each object, or assume rigid motion. We use a local geometric contact with the supported static surface.

Two papers set constraints on the design. MoDGS finds that the absolute depth values of foundation models drift from frame to frame, and so uses only ordinal (closer/farther) depth constraints. We therefore use foundation priors, at most, for an object's internal shape and never for its scale. USplat4D shows that monocular uncertainty is much larger along the viewing ray than across the image. Contact anchoring targets exactly that direction.

Core Claim

Free per-object motion leaves a moving object's depth undetermined up to a single unknown multiplier. We pin that multiplier with a contact loss. Let $\mathcal{C}_k$ be the contact pixels of moving region $k$, meaning the pixels on its lower boundary whose neighbor $p+\Delta$ directly below is confidently static. Let $D$ be predicted depth, and let $\mathrm{sg}$ denote stop-gradient. The loss is

$$ \mathcal{L}_{\text{contact}} = \sum_k \frac{1}{|\mathcal{C}_k|}\sum_{p \in \mathcal{C}_k} w(p)\,\big|\log D(p) - \mathrm{sg}\big[\log D(p+\Delta)\big]\big|, $$

where $w(p)$ down-weights contacts whose background neighbor has high warping error.

Inside each moving region, depth is shaped by the network's single-frame prediction. Optionally, an ordinal term from a depth foundation model can also shape it, in the manner of MoDGS. The contact loss only fixes where along the viewing ray the object sits.

Because the loss needs only contact, not rigidity, a walking pedestrian is treated exactly like a car. A built-in check also exists. On stationary objects such as parked cars and standing people, ordinary warping already gives a valid depth, so the contact-predicted depth can be compared against it during training. This lets us measure how well the anchor works on data we never use for evaluation.

Research Program

The base system is a Monodepth2-style network trained on Cityscapes and KITTI video. The first step is to find moving regions without a segmenter. We use ProDepth's disagreement between the multi-frame and single-frame predictions, combined with Monodepth2's auto-mask. A segmenter-based variant serves as a controlled comparison. The second step is to extract contact pixels as the lowest boundary of each connected moving region whose lower neighbor is low-uncertainty background. The third step adds the contact loss to the standard warping loss, which still applies only on static pixels.

We will study three failure regimes explicitly. The first is hidden contacts, such as a car whose wheels are blocked by another car. For these, the loss is switched off and the region falls back to the single-frame prediction. We report how often this happens. The second is objects supported by other moving objects, such as a rider on a bicycle. Here we anchor the rider to the bicycle and then the bicycle to the road. The third is shadows at the contact line. Shadows violate brightness constancy, a limitation that RobustSplat also reports. We measure how much contact detection degrades on shadowed contacts.

Baselines

The naive reference point is Monodepth2 with auto-masking, which supervises nothing on moving objects. It reports 0.115 AbsRel on the KITTI Eigen split. We retrain it with our exact backbone, data and schedule so that the gain from the added loss is isolated.

Insta-DM is the rigid joint-motion competitor, and it uses the camera-height trick. Its reported number is dynamic-region AbsRel of 0.178 on KITTI, down from 0.396 for its baseline. We run it with its released preprocessing on our splits.

The strongest competitor on the exact question of dynamic-region scale is Mining Supervision's depth-scale-alignment pipeline. Its reported result is a 52.6% Sq Rel reduction on Cityscapes dynamic regions. That figure is relative. The file gives no absolute dynamic-region value, so we must reproduce this baseline ourselves before claiming any margin over it. Beating it means lower dynamic-region AbsRel and Sq Rel on Cityscapes at matched static-region accuracy. It also means a clear margin on the non-rigid subset.

The strongest overall method is ProDepth, which reports Cityscapes AbsRel improving from 0.103 to 0.095 and KITTI AbsRel of 0.086 among multi-frame methods. Our method must match that whole-image number while improving on moving regions. Since we build on ProDepth's moving-region detection, the cleanest comparison is ProDepth with and without the contact loss.

A further baseline takes a depth foundation model's relative depth, fits one scale and shift per frame on static pixels, and uses the result as pseudo-labels on moving regions.

The ablations isolate the key ingredient. We remove the contact loss, remove the stop-gradient, and replace local contact with a single fitted ground plane, which approximates the camera-height assumption. We also swap uncertainty-derived masks for segmenter masks. Every method gets the same training data, image resolution, number of iterations and hyperparameter search budget.

Why not just…?

Why not just use learned scale alignment as in Mining Supervision? It is the closest competitor and may tie on cars. The difference shows on articulated movers. That pipeline assumes each object is rigid and needs separate networks trained in sequence. Contact anchoring trains in a single pass and assumes nothing about rigidity. If our method does not beat it on the pedestrian and cyclist subset, the claim fails, and we will report that outcome.

Why not just align a depth foundation model to the static background and distill it? This is the real threat. A single-image model has no confusion between motion and depth. MoDGS shows its absolute values drift over time, but alignment within each frame might absorb that drift. Global per-frame alignment still assumes the model's object-versus-background depth relationship is affine across the whole image, which nothing guarantees. Our experiment measures the gap between this baseline and contact anchoring on moving objects. If the gap is near zero, the honest conclusion is that foundation priors have made the contact constraint unnecessary when a foundation model is available. The method would then remain useful only in the setting with no foundation model.

Why not just use a camera-height or ground-plane constraint? Insta-DM's height constraint does pin scale, but only where the ground is a single known plane. The fitted-plane ablation tests this directly. We expect local contact to win on slopes, curbs and sidewalks, and to tie on flat roads.

Why not just auto-mask? Auto-masking avoids corrupted gradients but learns nothing on movers. Insta-DM's 0.396 dynamic-region baseline shows how large the resulting error is.

Proposed Experiments

The decisive test is dynamic-region AbsRel and Sq Rel on Cityscapes, using the dataset's standard depth evaluation as in the cited works. We split the results by class into rigid movers (cars, buses) and non-rigid movers (pedestrians, cyclists), and report whole-image metrics alongside them so static quality is visibly preserved. KITTI Eigen-split dynamic regions serve as a second test, with the caveat that moving objects are sparse there.

A key diagnostic is the error of contact-predicted depth on stationary objects in held-out Cityscapes video, measured against warping-derived depth. If the anchor is inaccurate on parked cars, it cannot be trusted on moving ones.

We will also evaluate zero-shot transfer to Waymo Open, following ProDepth's protocol. No numerical result for this protocol was found on Lacuna, so this comparison will rely on our own reruns.

The proposal succeeds if three things hold. It must beat the reproduced scale-alignment pipeline on non-rigid dynamic regions. It must match ProDepth's whole-image accuracy. And it must beat the foundation-alignment baseline by a margin larger than the variation across random seeds.