Most self-supervised systems that learn to see in 3D follow a simple, elegant logic: if we know how a camera moved between two frames, we can use geometry to "reproject" pixels from one frame to another. If the predicted depth is correct, the reprojected image should perfectly match the actual second frame. This "view synthesis" as supervision, pioneered by Tinghui Zhou and colleagues in Unsupervised Learning of Depth and Ego-Motion from Video, allows models to learn 3D structure from raw video without expensive LiDAR or manual labels.
However, this logic relies on a "static world assumption." It assumes that the only thing moving in the scene is the camera itself. In the real world—especially in autonomous driving—this assumption is constantly violated by cars, pedestrians, and cyclists.
The Problem of Hallucinated Depth
When an object moves independently of the camera, the geometric math breaks. As Ziyue Feng explains in Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth, standard algorithms treat these moving objects as noise or outliers.
The intuition for the failure is straightforward: if a car is driving at the same speed as your camera, its position in the image frame doesn't change. To a basic geometric solver, an object that doesn't move relative to the camera appears to be at infinite distance. This leads to "hallucinated" depths where moving vehicles are rendered as holes in the 3D map or assigned impossible distances, a phenomenon highlighted by Yihong Sun and Bharath Hariharan in Dynamo-Depth.
Decomposing the Scene into Independent Actors
To fix this, researchers have moved from masking moving objects to modeling them. Instead of treating a moving car as a region to ignore, the scene is decomposed into a global "ego-motion" (the camera's movement) and several local "object motions."
In Depth Prediction without the Sensors, Soren Pirk and Vincent Casser introduced a framework that uses instance segmentation to identify potential moving entities. Each detected object is assigned its own 3D motion vector. By treating the car as a separate actor with its own translation and rotation, the model can explain the pixel shifts as a combination of camera movement and car movement, preserving the correct depth.
This "instance-aware" approach is further refined by Qi Dai in Self-supervised Object Motion and Depth Estimation from Video, where objects are modeled as rigid bodies. By forcing the model to find a single 6-DOF (six degrees of freedom) transformation for an entire car rather than independent pixel movements, the system gains a much stronger geometric prior.
Holistic Motion Parsing
The most advanced versions of this idea, such as the "Holistic 3D Motion Parser" (HMP) proposed by Zhenheng Yang and Peng Wang in Every Pixel Counts, integrate depth, optical flow, and camera pose into a single unified framework.
The intuition here is cross-task consistency. As Yuliang Zou argues in DF-Net, depth and optical flow are two sides of the same coin. If you know the depth and the camera motion, you can predict the "static" part of the optical flow. Any remaining flow must be caused by independent object motion. By forcing these different predictions to agree with one another, the model learns to distinguish between the background and dynamic agents more effectively.
Beyond Rigid Bodies
While many models assume objects move as rigid blocks, recent work has pushed into more complex deformations. Haofei Xu and colleagues introduced Region Deformer Networks (RDN), which use bicubic deformation models to handle non-rigid objects like walking pedestrians. Similarly, Segment then Splat by Yiren Lu applies this decomposition to 3D Gaussian Splatting, representing each segmented object as a separate set of primitives that can be tracked and moved independently.
By shifting the paradigm from "ignoring motion" to "modeling actors," these researchers have transformed dynamic objects from a source of error into a rich source of geometric information. The result is a generation of vision systems that can perceive 3D space in busy urban streets as accurately as they do in static rooms.