Abstract
Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the-shelf recurrent networks, which only indirectly make use of the geometric information that is inherently available. We propose ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available. Taking inspiration from multi-view stereo, we propose a deep end-to-end cost volume based approach that is trained using self-supervision only. We present a novel consistency loss that encourages the network to ignore the cost volume when it is deemed unreliable, e.g. in the case of moving objects, and an augmentation scheme to cope with static cameras. Our detailed experiments on both KITTI and Cityscapes show that we outperform all published self-supervised baselines, including those that use single or multiple frames at test time.
Introduction
Estimating depth from a single image is a fundamental challenge in computer vision with significant implications for robotics and augmented reality. While monocular depth estimation has seen rapid progress, it remains inherently limited by the lack of geometric triangulation. Conversely, Multi-View Stereo (MVS) techniques leverage multiple frames to triangulate depth but often rely on expensive ground-truth supervision and struggle in dynamic environments where cameras are static or objects move independently.
ManyDepth addresses this "performance gap" by introducing an opportunistic framework that utilizes multiple video frames at test time when available, while gracefully falling back to single-frame estimation when necessary. Unlike previous methods that rely on slow test-time refinement, ManyDepth operates in a single forward pass, making it highly efficient for real-time applications.
Methodology
ManyDepth builds upon the Monodepth2 framework, extending it with a deep end-to-end cost volume. This structure matches features across consecutive frames to identify the most probable depth through geometric consistency.
Adaptive Cost Volumes and Scale
In self-supervised settings, the absolute scale of a scene is typically unknown. ManyDepth overcomes this by learning minimum and maximum depth bounds ($d_{min}, d_{max}$) directly from the data during training. This allows the cost volume to adapt dynamically to the network's internal scale, ensuring robust feature matching across different environments.
Consistency Loss and Motion Masking
A primary failure mode for cost-volume-based methods is the presence of moving objects, which violate the static-world assumption required for triangulation. To mitigate this, ManyDepth employs a monocular "teacher" network to provide a sanity check. A novel consistency loss compares the cost volume's predictions with the monocular network's output. If the predictions diverge significantly—often the case with moving cars or pedestrians—a binary motion mask $M$ is generated, instructing the network to ignore the unreliable cost volume and rely on monocular features instead.
Handling Static Cameras
MVS-style networks often fail when the camera is stationary (e.g., at a red light) because there is no baseline for triangulation. ManyDepth addresses this through a specific augmentation scheme during training, where the network is occasionally presented with identical frames or zeroed-out cost volumes. This teaches the model to maintain high-quality depth predictions even in the absence of camera motion.
Experimental Results
The model was evaluated on the KITTI and Cityscapes benchmarks, representing standard and complex urban driving scenarios, respectively.
- State-of-the-Art Performance: ManyDepth outperformed all published self-supervised baselines at the time of publication. On KITTI, it achieved a significant reduction in Absolute Relative (Abs Rel) error, moving from 0.115 (Monodepth2) to 0.098.
- Qualitative Superiority: The model excels at resolving thin structures and ambiguous, dark regions where single-frame models typically produce blurry artifacts.
- Efficiency: By utilizing a single forward pass rather than iterative refinement, ManyDepth is approximately 100 times more computationally efficient than competing high-accuracy multi-frame methods.
Conclusion
ManyDepth demonstrates that self-supervised monocular depth estimation can be significantly improved by "opportunistically" leveraging temporal information. By combining the geometric precision of cost volumes with the robustness of monocular recognition through a novel consistency loss, the model provides a scalable and efficient solution for dense depth estimation in real-world, dynamic environments.
Summary updated