The goal of monocular depth estimation is to predict a dense 3D map of a scene from a single 2D image. While humans do this effortlessly using contextual cues, machines traditionally required massive amounts of LiDAR ground truth to learn this mapping. As noted in Digging Into Self-Supervised Monocular Depth Estimation, this is an "ill-posed" problem that researchers have recently begun to solve without manual labels by using self-supervision.
The core "trick" of self-supervised depth estimation is treating depth as an intermediate step in a reconstruction puzzle. If a model can predict the depth of a frame and the motion of the camera, it can mathematically "warp" one video frame to look like the next. If the warped image matches the actual next frame (photometric consistency), the depth prediction was likely correct.
However, this elegant logic relies on a fragile premise: the static world assumption.
The Fragility of the Static World Assumption
Most self-supervised frameworks assume that the only thing moving in the video is the camera. Under this assumption, every pixel's change in position is purely a function of its depth and the camera's ego-motion.
As Rares Andrei Ambrus and Adrien Gaidon point out in SuperDepth, real-world driving scenarios rarely satisfy this. When a car drives at the same speed as the camera, it appears stationary in the video. The model, desperate to minimize reconstruction error, concludes that the car must be at an infinite distance because it isn't moving relative to the lens. This "infinite depth" artifact is a recurring failure mode across the field.
Furthermore, Matan Goldman and Tal Hassner observe that independent motion—like a pedestrian crossing the street—violates the epipolar geometry used for training. This leads to "shadow" artifacts and blurred boundaries where the model fails to reconcile the object's movement with the background's geometry.
Reconciling Monocular and Multi-View Cues
To solve this, researchers are moving away from pure geometric reconstruction toward a hybrid approach that blends monocular cues (contextual "priors" about what objects look like) with multi-view cues (geometric triangulation).
In Adaptive Fusion of Single-View and Multi-View Depth for Autonomous Driving, Junda Cheng and Wei Yin introduce AFNet, which uses confidence maps to decide which cue to trust. In static regions, the model relies on multi-view geometry; in dynamic regions where geometry fails, it pivots to monocular context. Similarly, Dong Gong and colleagues proposed a Cross-Cue Fusion (CCF) module in Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth Estimation in Dynamic Scenes, which reduced errors in dynamic areas by over 20% by intelligently blending these signals without needing hand-crafted masks.
Semantic Guidance and Motion Modeling
Another powerful strategy involves teaching the network what is moving. Vitor Guizilini and the team at Toyota Research Institute demonstrated in Semantically-Guided Representation Learning for Self-Supervised Monocular Depth that semantic information can "guide" the network. By identifying pixels belonging to "car" or "pedestrian" categories, the model can apply specific constraints to those regions, preventing them from being projected to infinity.
Some researchers take this a step further by explicitly modeling the 3D motion of objects. In Self-supervised Object Motion and Depth Estimation from Video, Vaishakh Patil and Simon Hecker treat moving objects as rigid bodies with their own 6-DoF motion. By solving for both camera motion and object motion simultaneously, the geometric "contradiction" is resolved, allowing for accurate depth even in chaotic traffic.
From Static Benchmarks to Casual Video
The evolution of this field is a transition from controlled datasets like KITTI to "in-the-wild" scenarios. SC-DepthV3 by Libo Sun uses "imperfect" pseudo-depth from pre-trained models as a structural prior to stabilize training in highly dynamic environments.
This shift is perhaps best exemplified by Structure and Motion from Casual Videos, which enables 3D reconstruction from standard smartphone videos. By jointly optimizing depth, camera pose, and a per-pixel "movement map," these systems are finally overcoming the static-world limitations that once confined self-supervised depth estimation to empty roads and stationary backgrounds.