Self-supervised depth networks learn 3D structure from ordinary video. They predict a depth map and a camera motion, use the two to re-draw one frame from its neighbor, and are trained to make that re-drawing match the real image. The approach rests on the assumption that the world is static. A moving car breaks it, and the network then either distorts the car's depth or contaminates the camera-motion estimate. Current fixes run two separate pieces of machinery. A learned network regresses camera motion directly from pixels, and a separate heuristic mask throws away pixels whose re-drawing error looks too large. The two never communicate, even though they answer the same question: which pixels move consistently with the camera?
We propose to make them one computation. Inside training, camera motion over a short window of frames is solved by a differentiable robust estimator, a solver that iteratively down-weights measurements that disagree with the consensus. It works on dense correspondences built from predicted depth and a jointly learned, rigidity-free optical flow (per-pixel 2D motion between frames). The weights the solver assigns to each pixel are the motion mask. Because a short window yields many frame triplets, the estimate can be checked for closure around loops of frames. That check separates "this pixel moves independently" from "this pixel's depth is merely wrong" or "this pixel is occluded or shiny." Each of those three cases then gets its own treatment, instead of all of them being thrown away together.
Core Claim
The claim is that the robust weights of a window-level camera-motion solver give a better self-discovered motion mask than photometric thresholds, and a better camera motion than a regressed pose network, on scenes with many moving objects. Using those weights to route pixels gives supervision on moving objects without any external segmentation or depth model. If the claim holds, depth error on moving-object pixels in Cityscapes and DDAD should fall clearly below that of the strongest purely self-supervised mask method. Static-scene accuracy should be no worse.
Five lines of prior work frame this.
The first motivates the proposal. On the Sins of Image Synthesis Loss shows that photometric loss can keep falling while depth error rises, with rank correlation collapsing or turning negative in 54% of the cases it tested. That makes photometric residuals a poor basis for deciding what is an outlier. We decide on geometric residuals instead.
The second is the closest existing work, which consists of loss-level dynamic masks. D³epth flags pixels whose reprojection error exceeds an adaptive quantile in both neighboring frames and discards them. Manydepth2 uses flow to build a pseudo-static frame for its cost volume but still relies on a motion-guided mask during training. Both keep the pose estimate separate from the mask, and both leave masked pixels unsupervised. We tie the two together and supervise the masked pixels.
The third makes moving objects explicit but leans on external models. Self-supervised Object Motion and Depth Estimation gives each object its own rigid 6-degree-of-freedom motion using Mask R-CNN instances. SC-DepthV3 supervises dynamic regions with ordinal constraints from a pretrained depth model. We keep the idea of a per-region rigid fit but discover the regions from solver residuals, with no pretrained segmenter or depth prior.
The fourth supplies the method. Cycle-Sync reweights edges of a camera graph by how well 3-cycles close and anneals a Welsch-type loss, which is linear near zero and flat for large errors. Revisiting Rotation Averaging marginalizes over noise thresholds with MAGSAC++ instead of fixing one threshold. Both operate offline on sparse matches. Cycle-Sync is also known to fail on sparse or near-linear camera graphs, which is exactly what a driving trajectory looks like. Within a 5-frame window, however, we can solve all pairwise motions, so the graph is complete and 3-cycles are plentiful. This sidesteps that limitation.
The fifth is SPIdepth. It shows that pose quality is the main bottleneck in this family of methods, and it addresses the problem by scaling the pose network. We test whether replacing regression with a robust solve beats scaling.
Research Program
The first component is the window solver. For a window of $N$ frames (we use $N=5$), a depth network predicts a depth map for each frame. A flow head, trained with an ordinary photometric loss and no rigidity constraint, predicts forward and backward flow for every frame pair. Lifting each pixel to 3D with its predicted depth, and following its flow into the other frame's predicted depth, gives a dense set of 3D-to-3D correspondences per pair. Each pairwise camera motion $T_{ij}$ is then solved by annealed iteratively reweighted least squares under a Welsch loss, initialized from a small pose network. The pairwise motions are synchronized over the window with cycle-consistency reweighting in the style of Cycle-Sync.
Gradients pass through the fixed point of the solver by implicit differentiation, so the depth and flow networks learn to produce correspondences that the solver finds consistent. The per-pixel weight $w_p \in [0,1]$ at convergence, aggregated over all pairs containing that pixel's frame, is the static-confidence map. The camera motion used for warping comes from the solver, not from the pose network.
The second component is three-way routing. A pixel is treated as static if it has high weight under the synchronized motions. It is treated as independently moving if it has low weight but its flow is forward-backward consistent and its residual repeats consistently across pairs. It is treated as unreliable if its flow itself is inconsistent, which covers occlusion, specularity and textureless regions.
Static pixels receive the standard photometric loss plus a geometric depth-consistency loss. Unreliable pixels are dropped. Moving pixels are grouped by clustering their residual motion, and each group gets its own robust rigid fit. The group's depth is then supervised by consistency along its flow track under that fitted motion, borrowing the geometric constraint idea of Dai et al. to stop the fit from collapsing to zero motion. Non-rigid movers such as pedestrians produce poor per-group fits and receive low weight automatically rather than wrong supervision.
There is one failure we expect and do not claim to solve: a vehicle moving at exactly the camera's speed. It has zero flow, and a depth at infinity makes it fully consistent with the camera motion, so the solver will accept it as static (the "infinite depth" failure described by Guizilini et al.). We will report this slice separately rather than hide it.
Baselines
The naive reference is Monodepth2 with minimum reprojection and auto-masking, retrained in our codebase. On Cityscapes it is reported at AbsRel 0.129 in the D³epth paper, and on KITTI at AbsRel 0.119 per the MonoViT table.
The strongest purely self-supervised competitor is D³epth, reported at AbsRel 0.097 on Cityscapes. It is the most direct rival because it attacks the same failure: moving objects corrupting both neighbors. It does so with a photometric quantile mask, while we use solver weights. We will retrain it with the same depth backbone, resolution, epochs and per-method tuning budget. Beating it means lower AbsRel on Cityscapes overall and, more importantly, lower error restricted to moving-object pixels.
The strongest method that uses outside help is SC-DepthV3, reported at AbsRel 0.142 on DDAD, which relies on a pretrained depth prior. We compare against it on DDAD and on the dynamic splits of Bonn and TUM. Matching it without any external model would count as success; we do not require beating it.
SPIdepth, with a large pretrained pose network, isolates whether a better pose regressor alone explains any gains. It reports AbsRel 0.029 on KITTI, and on Cityscapes only a relative improvement over SQLdepth (no absolute Cityscapes number found on Lacuna). We will run it with the same depth network as ours.
Manydepth2 is the flow-based alternative. On Cityscapes it reports only an approximately 15% improvement over ManyDepth, with no absolute number found on Lacuna.
Three ablations isolate the key ingredient:
- Solver kept, mask removed: the solver's camera motion is used but its weights are not used as the mask; D³epth's quantile mask is used instead.
- Mask kept, solver removed: the solver's weights are used as the mask, but warping uses the pose network's motion.
- Loop check removed: cycle reweighting is removed and pairs are solved independently.
A fourth ablation removes the moving-object supervision, so moving pixels are dropped rather than routed. This separates the gain from better masking from the gain from supervising moving pixels.
Why not just…?
Why not just combine D³epth's mask with SPIdepth's large pose network? This is the most serious threat, because it may capture most of the gain cheaply. The ablation "solver kept, mask removed" and a direct D³epth-plus-SPIdepth run settle it. If that combination matches us on moving-pixel error, the solver adds nothing, and we would report that. Our reason to expect otherwise is that a regressed pose sees moving pixels as input and has no mechanism to ignore them, while a robust solve explicitly down-weights them.
Why not just put a Huber or Welsch loss on the photometric error? A robust loss on the photometric error still trusts photometric residuals, which the Sins paper shows are misaligned with geometric error. It also needs a fixed threshold, which Revisiting Rotation Averaging shows to be fragile (AUC@5° of 64.6 versus 54.7 for fixed-loss baselines on 1DSfM). We include a tuned-Welsch-photometric baseline. If it closes the gap, the geometric-residual part of our claim fails.
Why not just mask moving classes with Mask R-CNN? Class masks remove parked cars, which are valuable static constraints. They also miss moving shadows, as 3DGS-HPC notes, and they leave masked regions unsupervised, which SC-DepthV3 identifies as the cause of blurry moving-object depth. We run this baseline anyway, as a semantics-assisted reference.
Why not just add a pose cycle-consistency loss to the pose network? A cycle loss on the pose network's outputs makes poses mutually consistent, but it can be satisfied by consistently wrong poses. It also produces no per-pixel weights. The "loop check removed" ablation and a pose-network-plus-cycle-loss variant test whether closure matters only at the pose level or also through the pixel weights.
Proposed Experiments
We train on KITTI (Eigen split), Cityscapes and DDAD, and evaluate depth with AbsRel, SqRel, RMSE, RMSE log and the δ thresholds. Results are reported both over all pixels and over moving-object pixels.
Moving-object pixels come from different sources per dataset. KITTI-2015 scene flow supplies object maps. On Cityscapes we define moving pixels as instances of movable classes that our evaluation flags as moving relative to the static background; these masks are used only for evaluation. DDAD and Bonn/TUM are used for long-range and indoor dynamics.
The solver weights are scored as a motion mask against KITTI-2015 object maps (IoU) and compared with D³epth's mask. Camera motion is evaluated on KITTI odometry. Because KITTI is about 0.34% dynamic pixels, we expect parity there; the decisive tests are on Cityscapes and DDAD.
Three checks decide whether the idea holds.
The first is whether moving-pixel AbsRel on Cityscapes falls meaningfully below a retrained D³epth while all-pixel AbsRel stays at or below its reported 0.097.
The second is whether the solver weights achieve higher motion-mask IoU than the quantile mask without wrongly marking parked cars as moving. We measure this as the false-positive rate on stationary vehicles.
The third repeats the divergence analysis of the Sins paper. We compute the rank correlation between training loss and held-out depth error across training for our method and for Monodepth2. If geometric routing is doing its job, our correlation should stay positive where the baseline's collapses.
The co-moving-vehicle slice is reported separately as a known limitation.
References
- On the Sins of Image Synthesis Loss for Self-supervised Depth Estimation
- Depth: Self-Supervised Depth Estimation with Dynamic Mask in Dynamic Scenes
- Manydepth2: Motion-Aware Self-Supervised Monocular Depth Estimation in Dynamic Scenes
- Self-supervised Object Motion and Depth Estimation from Video
- SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes
- Cycle-Sync: Robust Global Camera Pose Estimation through Enhanced Cycle-Consistent Synchronization
- Revisiting Rotation Averaging: Uncertainties and Robust Losses
- SPIdepth: Strengthened Pose Information for Self-supervised Monocular Depth Estimation
- Semantically-Guided Representation Learning for Self-Supervised Monocular Depth
- MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer