When a network learns depth from unlabeled video, its only teacher is the assumption that the world stands still. It predicts depth and camera motion, warps one frame into the next, and is penalized wherever the warp fails to reproduce the image. Moving people, animals, cyclists and deforming tissue break this assumption. Current methods respond in one of two ways. Some throw those pixels away as outliers. Others assume everything deforms smoothly and regularize the whole frame uniformly. Neither approach can say which pixels' motion is actually known. As a result, neither can fill in motion where it is ambiguous, such as on textureless shirts, at occlusion edges, or on specular skin.
We propose to treat "this pixel obeys the static world" as a soft, per-pixel hypothesis with a measurable confidence, and to use that confidence to steer spatial propagation. Confidence-guided propagation means letting reliable pixels overwrite unreliable neighbors instead of averaging everyone together. The new choice is what gets propagated. We spread the residual 3D motion field (the part of each pixel's motion not explained by the camera), and depth is then recovered through motion-compensated photometric loss. Confidently static pixels pin their residual to zero. Confidently tracked moving pixels pin it to their measured value. Everything else is filled in by learned affinities that are blocked from crossing boundaries where rigidity confidence changes sharply. In two sentences: rigid background and deforming actors are separated by a confidence map rather than by a mask, and that same map decides which pixels' motion is trusted and spread to their neighbors. This gives dense non-rigid motion in ambiguous regions without semantic labels, LiDAR, or ground-truth depth.
Core Claim
In self-supervised monocular depth for dynamic, non-rigid scenes, error concentrates on moving actors and inside ambiguous regions. Both are better handled by confidence-gated propagation of the residual motion field than by any of three alternatives: masking those pixels out, applying uniform smoothness or as-rigid-as-possible (ARAP) priors, or using the same confidence only to down-weight the loss. Two conditions are necessary. The confidence must be built to resist the "confident but wrong" failure, and propagation must act on motion rather than on depth.
Where This Sits
Three lines of self-supervised dynamic depth motivate the gap. The first treats non-rigid motion as residue. Unsupervised Monocular Depth Learning in Dynamic Scenes learns a 3D translation field regularized by an $L_{1/2}$ sparsity loss and group smoothness. RM-Depth adds recurrent residual motion warping with an outlier-aware loss. Both restrict motion to translation and let non-rigid pixels drop out, which is exactly the information we want to keep. The second line models deformation but without reliability. Unsupervised Monocular Depth Reconstruction of Non-Rigid Scenes applies an ARAP loss over Euclidean distance matrices uniformly across the frame. Region Deformer Networks fits a 32-coefficient bicubic warp per object but needs Mask R-CNN masks. The third line, feed-forward foundation depth such as Depth Anything, gives strong per-frame relative depth but no motion. It also flickers across frames, as DVD notes.
The propagation machinery comes from depth completion and refinement. Learning Affinity via Spatial Propagation Networks showed that learned linear propagation solves anisotropic diffusion in linear time. NLSPN put confidence into affinity normalization with non-local neighbors. DySPN made affinities dynamic and converged in 6 iterations. Masked SPN grows a reliable region outward from anchors. All of these propagate depth from LiDAR anchors. We reuse their mechanism with anchors that come from geometric self-consistency, and we apply it to a motion field.
One result constrains the design. Probabilistic Pixel-Adaptive Refinement Networks found that confidence-weighted refinement spreads errors when confidence is high on wrong values. In dynamic video, the canonical confidently wrong case is the "infinite depth" trap identified by Region Deformer Networks: an object moving with the camera is explained perfectly by the static model if it is assigned infinite depth. Self-Supervised Monocular Depth Hints arbitrates between hypotheses using photometric error. That arbitration is itself fooled by this trap, so our confidence cannot rely on photometric error alone.
Research Program
The model has three parts: a depth network, a camera-motion network, and a residual-motion head that outputs a 3D vector $m_i$ per pixel $i$. A per-pixel rigidity confidence $c_i \in [0,1]$ is computed from two signals, combined multiplicatively.
The first signal is multi-frame static consistency. It asks whether the static reprojection explains pixel $i$ across a window of several frames, both forward and backward, rather than in a single frame pair. This matters because a moving object rarely satisfies static geometry consistently in both temporal directions.
The second signal is ordinal agreement with a frozen relative-depth prior from Depth Anything. Pixel $i$ loses confidence when its predicted depth order relative to its neighbors contradicts the prior's order. This targets the infinite-depth trap without instance masks: a pedestrian assigned depth beyond the road behind them violates the prior's ordering. It does so without ever copying the prior's metric values.
A separate anchor set marks moving pixels whose motion is known. These are pixels where a pretrained optical flow passes forward-backward cycle consistency and the flow is not explained by the static model. Their residual motion is triangulated from flow plus depth.
Propagation then runs a few iterations of confidence-gated non-local updates on the residual field, in the style of NLSPN and DySPN:
$$m_i^{(k+1)} = \alpha_i\, \hat m_i + (1-\alpha_i) \sum_{j \in \mathcal{N}(i)} \frac{a_{ij}\, \alpha_j}{\sum_{l \in \mathcal{N}(i)} a_{il}\, \alpha_l}\, m_j^{(k)}$$
The symbols are as follows. $\hat m_i$ is the anchor value: zero for confidently static pixels, the triangulated motion for confident moving anchors, and the head's raw prediction otherwise. $\alpha_i$ is the anchor strength: $c_i$ for static anchors, the cycle-consistency score for moving anchors, and near zero elsewhere. $\mathcal{N}(i)$ is a learned set of non-local neighbors. $a_{ij}$ is a learned affinity computed from image features and from the difference $|c_i - c_j|$. Feeding in that difference lets the network learn not to spread motion across rigidity boundaries, which is the contact-boundary entanglement problem raised by Split4D.
The propagated field warps 3D points before reprojection. The photometric loss therefore trains depth through motion-compensated geometry everywhere, including on actors that prior methods mask out. A light local-rigidity term on $m$ is applied only within high-affinity neighborhoods. It is not applied frame-wide, unlike the uniform ARAP prior of the distance-matrix method.
The analysis has three parts. We measure how well $c_i$ is calibrated against ground-truth motion masks. We measure how often moving objects end up farther than the background behind them, as a direct count of the infinite-depth trap. We report the cost relative to the base network, since PPAC reported roughly 1.7x overhead and DySPN's few-iteration design is the template for keeping this small.
Baselines
The naive reference is a self-supervised monocular network with a static scene model, auto-masking and edge-aware smoothness only. It is the same backbone with the motion head and propagation removed. Every gain is measured relative to it.
The strongest existing self-supervised dynamic method in the file is RM-Depth. It reports Abs Rel 0.107 on KITTI and outperforms PackNet at 0.111. It is the right competitor because it already learns residual motion without labels and differs from us mainly in masking non-rigid outliers. We will run its released configuration and also retrain it on our backbone, data, resolution and epoch budget, giving both the same hyperparameter search. Matching 0.107 on whole-image KITTI Abs Rel while clearly reducing error on dynamic-object pixels would count as beating it. KITTI's full-image metric is dominated by static road, and we do not expect large global gains there.
On non-rigid data, the competitor is the distance-matrix ARAP method in Unsupervised Monocular Depth Reconstruction of Non-Rigid Scenes. It reports Abs Rel 0.519 on VolumeDeform versus 0.774 for PackNet, and motion-segmentation mIoU 0.731 on Sintel. We will compare on both, using the same flow network for correspondences so that flow quality is not a confound. Beating it means lower VolumeDeform Abs Rel and higher Sintel mIoU when our thresholded confidence is read as a segmentation.
The translation-sparsity method, Unsupervised Monocular Depth Learning in Dynamic Scenes, serves as the existing regularizer baseline. No reported KITTI or Cityscapes number for it was found in the file, so it will be run ourselves under the shared protocol.
Three ablations isolate the key ingredients, each sharing everything except the named component. The first uses the identical confidence only as a per-pixel loss weight, with no propagation. The second propagates depth instead of residual motion, with the same confidence and affinities. The third drops the ordinal prior term from the confidence.
Why not just…?
Why not just use the confidence to down-weight the loss, as uncertainty-attenuated training does? This is the most serious threat. Learning Stereo Matchability and Yoon et al. show that attenuation alone helps. Attenuation stops bad gradients but supplies no motion where none is observed, so we predict it ties our method on static and textured regions and loses in textureless moving regions. The loss-weight-only ablation tests exactly this. If it matches full propagation on dynamic-region error, propagation is not earning its cost and the claim fails.
Why not just run Depth Anything per frame, align scale per frame, and add optical flow? This could win on single-frame depth metrics, and we will report it honestly. It does not produce a consistent 3D motion field. DVD documents its temporal flicker and scale drift, and Depth Anything concedes that the noise in its pseudo-labels remains unsolved. We therefore evaluate temporal depth consistency and scene-flow error on actors, not only Abs Rel. Our model uses the prior only ordinally, so if this baseline wins on all three measures, the geometric machinery is unjustified.
Why not just mask movers with an off-the-shelf segmenter and reconstruct the rest rigidly? Masking discards the actors' geometry, which is the output we care about. Split4D shows tracker-dependent decomposition degrading badly under fast motion and contact, with a Multi-Human mIoU of 0.893 for Split4D versus 0.592 for the tracking-based SA4D. Pretrained segmenters also do not transfer to surgical tissue in Hamlyn. A Mask R-CNN masking baseline on KITTI and Cityscapes will show whether segmentation buys anything over confidence on static-region accuracy.
Why not just smooth the motion field with edge-aware or group-sparsity priors? That is precisely the existing regularizer. Image edges do not coincide with rigidity boundaries: a person's limb and their textureless torso share color but not motion. The comparison against the translation-sparsity method and the depth-propagation ablation tests whether gating by confidence differences matters.
Proposed Experiments
Training uses only unlabeled monocular video: KITTI raw and Cityscapes for driving, VolumeDeform and Hamlyn for non-rigid scenes, and Sintel for analysis. Evaluation uses the KITTI Eigen split for Abs Rel, compared against RM-Depth's 0.107 and Region Deformer's 0.1377. We report errors split into static and dynamic pixels, using KITTI 2015 and Cityscapes instance labels at evaluation time only. VolumeDeform Abs Rel is compared against 0.519. Sintel motion-segmentation mIoU is compared against 0.731, and Sintel residual-motion error is also reported. For Hamlyn, no reported result was found on Lacuna, so we report qualitative results and photometric consistency only.
Three checks are decisive. The first is the loss-weight-only versus propagation ablation on dynamic-pixel error and on textureless dynamic subsets. The second counts moving objects predicted farther than the background directly behind them, with and without the ordinal prior. This tests whether the confidence actually defuses the confidently wrong trap rather than spreading it. The third is the calibration of $c_i$ against Sintel ground-truth motion. If rigidity confidence is poorly calibrated, the propagation story collapses regardless of headline numbers. If confidence is well calibrated but propagation adds nothing over loss weighting, we will report that as the finding.
References
- Unsupervised Monocular Depth Learning in Dynamic Scenes
- RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes
- Unsupervised Monocular Depth Reconstruction of Non-Rigid Scenes
- Region Deformer Networks for Unsupervised Depth Estimation from Unconstrained Monocular Videos
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
- DVD: Deterministic Video Depth Estimation with Generative Priors
- Learning Affinity via Spatial Propagation Networks
- Non-local Spatial Propagation Network for Depth Completion
- Dynamic Spatial Propagation Network for Depth Completion
- Masked Spatial Propagation Network for Sparsity-Adaptive Depth Refinement
- Probabilistic Pixel-Adaptive Refinement Networks
- Self-Supervised Monocular Depth Hints
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Learning Stereo Matchability in Disparity Regression Networks
- Leveraging stereo matching with learning-based confidence measures