When a network learns depth from ordinary video without labels, it learns by warping one frame into the next and penalizing color mismatches. This works for the static world. It fails for moving cars and pedestrians. A car driving at the same speed as the camera looks stationary in the image, so the network explains it most cheaply by placing it infinitely far away. Current systems avoid the damage by masking such regions out of the loss. They also mask out occluded pixels, the ones hidden in one frame and visible in the other. Both kinds of pixel are treated as noise. We argue that the occlusions are exactly the evidence needed to fix the moving objects. When a moving car covers a patch of road or building, that patch's depth is already pinned down by the camera's own motion. So the car must be in front of it. That is a metric upper bound on the car's depth, and it rules out the infinite-depth failure directly.
The proposal is a self-supervised depth-and-motion learner in which visibility is computed by a soft depth ordering over the surfaces that land on each pixel: the rigidly warped static background and the independently translated moving parts. It uses no semantic segmentation. Photometric agreement then decides which surface actually won each pixel, and gradients flow into the depths through that decision. Occlusion stops being a binary mask and becomes a supervisory signal. This turns the ordinal observation behind DynOcc, that moving boundaries reveal depth order, into a metric constraint inside a reconstruction model.
Core Claim
The ambiguity is this: from a single moving camera, a moving object's depth and its 3D velocity trade off along a one-parameter family that the object's own pixels cannot distinguish. Occlusion and disocclusion events at the object's silhouette break that tie. Those events involve background surfaces whose depth comes from static parallax. We claim that differentiable depth ordering over static and dynamic surfaces, with no class priors, will substantially reduce depth error on moving-object pixels relative to masking-based treatments. The gains should concentrate on the co-moving-object failure case, and the method should close most of the gap to systems that rely on a supervised segmentation network.
Where This Sits in Prior Work
Three lines of work supply the ingredients but stop short of the combination.
The first line derives occlusion masks from the model's own geometry, but only under camera motion. Depth From Videos in the Wild uses a differentiable Z-buffer, a test of which reprojected point is nearest along each ray. It allows residual motion only inside boxes from a pretrained detector, so unrecognized movers are treated as static. Wang et al. compute forward-backward masks from depth and pose. They discard dynamic regions with a less-than-mean error heuristic and assume hidden regions are static. GeCoNeRF builds depth-comparison occlusion masks for sparse-view radiance fields, but its hand-tuned threshold is exactly the kind of hard decision we replace with a soft ordering.
The second line models independent motion without semantics, but ignores visibility. Li et al. learn a sparse 3D translation field with an $L_{1/2}$ penalty and beat a segmentation-dependent baseline on Cityscapes. They do not model object-background occlusion and are known to degenerate when object and camera motion are collinear. We reuse their motion field as the source of "dynamic surfaces." What we add is the ordering that couples those surfaces to the background.
The third line uses semantics to do both. DynamicDepth disentangles object motion using a single-frame prior and an occlusion-aware reprojection loss. It depends on a supervised panoptic segmenter to decide which pixels may move. It is our strongest competitor and the target to match without that dependency.
One apparent contradiction is worth facing directly. RobustSplat and Wang et al. show that treating transient errors as signal corrupts static geometry. That argues against using moving boundaries. Our answer is that the corruption comes from explaining transient pixels with static geometry. We explain them with a separate moving surface whose depth is constrained, not by growing background structure. DynOcc's finding that 21 million dynamic ordinal pairs beat 223 million static ones is the motivating evidence that the signal is worth exploiting.
Research Program
The model has a single-frame depth network, a pose network, and a per-pixel 3D translation field regularized as in Li et al. For a target pixel $p$, the renderer collects every source pixel that reprojects onto $p$. Static source pixels are warped by camera motion alone, and pixels with nonzero translation are warped by camera motion plus their own translation. Each candidate $k$ carries a target-frame depth $z_k$ and a color $c_k$. Instead of a hard Z-buffer, the renderer blends candidates with visibility weights
$$w_k = \frac{\exp(-z_k/\tau)}{\sum_j \exp(-z_j/\tau)}, \qquad \hat{I}(p) = \sum_k w_k c_k,$$
where $\tau$ is a temperature annealed toward zero during training. The loss compares $\hat{I}(p)$ to the observed target color. Suppose a moving car's pixels and the static wall behind it both land on $p$, and the observed color matches the car. The loss can then only be lowered by making the car nearer than the wall. That is the constraint $z_{\text{car}} < z_{\text{wall}}$, with the wall's depth fixed by the static warp. Disocclusions give the complementary signal when the renderer runs in the reverse direction. Pixels with no incoming candidates fall back to the usual out-of-view handling.
The analysis component checks whether the gain comes from the ordering or from extra capacity. We will measure how often, on held-out data, predicted object depth violates the occlusion ordering implied by ground truth. We will also plot moving-object error against the fraction of an object's silhouette that participates in an occlusion event during the clip. If the mechanism is right, error should fall with occlusion coverage for our model and not for the masking baselines.
Baselines
The naive reference point is the same network trained under the static assumption, with the translation field disabled and no occlusion handling. This establishes how badly moving objects are hurt. The closest reported static-assumption number in the file is ManyDepth at Abs Rel 0.114 on Cityscapes, as reported by DynamicDepth.
The strongest existing method is DynamicDepth itself. It reports Cityscapes Abs Rel 0.103 overall and 0.129 on dynamic-object pixels, and 0.068 on KITTI with improved ground truth. We will retrain it with its released recipe at our resolution and with our backbone where feasible, and quote both our rerun and the paper's numbers. Beating it means matching its overall error while improving dynamic-pixel Abs Rel below 0.129, without any segmentation network. Matching it without semantics would itself be a meaningful result.
The semantic-free motion baseline is Li et al. For this we found no reported Abs Rel on Lacuna for Cityscapes, so we will retrain it under the identical data, schedule and tuning budget as our method. Depth From Videos in the Wild reports KITTI Abs Rel 0.128 and serves as the hard Z-buffer with semantic boxes reference.
The key ablation isolates the ingredient. It is Li et al.'s motion field plus a hard Z-buffer mask that excludes occluded pixels. Everything is identical except that occlusion is masked rather than rendered with soft ordering. If this ablation matches the full model, the claim fails. Every trained baseline gets the same number of hyperparameter trials and the same GPU budget.
Why not just…?
Why not just mask movers with an off-the-shelf segmenter, as DynamicDepth does? This works for the classes the segmenter knows, and DynamicDepth's numbers show it. Split4D documents that 2D segmentation and tracking drift at occlusions and fast motion. Gordon et al. note that unrecognized movers silently fall back to the static model. Masking also only removes the bad gradient; it does not tell the network where the object is. We test this directly: we run DynamicDepth and our model on moving objects outside the segmenter's taxonomy, identified from ground-truth scene flow on held-out sequences, and report dynamic-pixel error separately for in-taxonomy and out-of-taxonomy movers.
Why not just distill depth for dynamic pixels from a monocular foundation model? This is the most serious threat. Align3R shows that single-frame depth priors substantially improve dynamic-video depth (Sintel Abs Rel 0.253 versus 0.335 for MonST3R). Such a prior likely removes the infinite-depth failure on common road scenes. Its depth is relative per frame, however, and must be aligned in scale and shift to the self-supervised static geometry. The occlusion ordering supplies exactly that alignment on moving objects. We will train a prior-distillation variant and a variant with both the prior and the ordering loss. If the prior alone matches the combination on dynamic-pixel error, we will report that the contribution reduces to a fully self-supervised regime. We would not claim more than that.
Why not just use the hard Z-buffer with a motion field? This is the ablation above. A hard mask gives zero gradient to the ordering decision, so it cannot move a co-moving car forward from infinity. It can only stop penalizing it. The soft-ordering-versus-hard-mask comparison on the co-moving subset settles this.
Proposed Experiments
We will train on Cityscapes and KITTI raw video and evaluate on the Cityscapes test protocol used by DynamicDepth and on the KITTI Eigen and improved-ground-truth splits, reporting Abs Rel, squared relative error, and threshold accuracy. We will report separate numbers on dynamic-object pixels and on a co-moving subset: objects whose ground-truth image-plane displacement after ego-motion compensation falls below a small threshold. On this subset every baseline except the semantic one should fail badly. Ordinal accuracy across moving silhouettes, in the spirit of DynOcc's evaluation, gives a direct readout of whether the ordering is learned.
The decisive checks are three. First, the soft-ordering model must beat the hard-mask ablation on the co-moving subset by a margin well outside seed variance across three seeds. Second, the dynamic-pixel error must fall with occlusion coverage, as the mechanism predicts. Third, static-region error must not degrade relative to the naive baseline, which would signal the RobustSplat-style corruption we claim to avoid. If the first check fails, the idea is wrong. If only the third fails, the temperature schedule or the sparsity weight on the motion field needs revisiting before any broader claim.
References
- DynOcc: Learning Single-View Depth from Dynamic Occlusion Cues
- Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras
- Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry
- GeCoNeRF: Few-shot Neural Radiance Fields via Geometric Consistency
- Unsupervised Monocular Depth Learning in Dynamic Scenes
- Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Align3R: Aligned Monocular Depth Estimation for Dynamic Videos