Abstract
Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the-shelf recurrent networks, which only indirectly make use of the geometric information that is inherently available. We propose ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available. Taking inspiration from multi-view stereo, we propose a deep end-to-end cost volume based approach that is trained using self-supervision only. We present a novel consistency loss that encourages the network to ignore the cost volume when it is deemed unreliable, e.g. in the case of moving objects, and an augmentation scheme to cope with static cameras. Our detailed experiments on both KITTI and Cityscapes show that we outperform all published self-supervised baselines, including those that use single or multiple frames at test time.
Introduction
Predicting 3D depth from a single 2D image is a fundamental challenge for autonomous driving and augmented reality. While self-supervised methods have advanced significantly by learning from video sequences, they typically discard temporal context at inference time, relying solely on a single frame. ManyDepth addresses this inefficiency by "opportunistically" utilizing multiple frames at test time to enhance depth accuracy through a geometric cost-volume approach.
Methodology
ManyDepth builds upon a plane-sweep stereo architecture, introducing three innovations to adapt it for self-supervised monocular video:
Adaptive Cost Volumes
Standard Multi-View Stereo (MVS) requires predefined depth ranges. Since self-supervised models learn depth only up to an unknown scale, ManyDepth dynamically learns depth extents ($d_{min}$ and $d_{max}$) from the data during training using an exponential moving average of its own predictions.
Consistency Loss
MVS assumes a static world, but real-world driving sequences are full of moving objects. ManyDepth uses a separate monocular network as a "teacher" during training. If the cost volume's geometric matching disagrees with the monocular prediction, a consistency mask is triggered, forcing the model to ignore the unreliable cost volume and rely on monocular semantic cues instead.
Static Camera Augmentation
Geometric triangulation fails when the camera is stationary. To ensure robustness, the authors augment training by randomly replacing previous frames with static or color-augmented versions of the current frame, teaching the network to fall back on monocular cues when motion is absent.
Results and Discussion
ManyDepth outperforms all published self-supervised baselines on the KITTI and Cityscapes benchmarks. On the KITTI Eigen test set, it achieves an Absolute Relative Error (Abs Rel) of 0.087.
Beyond accuracy, ManyDepth is highly efficient. Unlike methods requiring expensive test-time optimization, it uses a single forward pass, making it roughly 100x faster and suitable for real-time applications.
Conclusion and Future Work
By intelligently combining geometric MVS with monocular semantic reasoning, ManyDepth provides a robust framework for multi-frame depth estimation. While it remains dependent on accurate pose estimation and fixed camera intrinsics, its ability to switch between geometric and semantic cues makes it a robust solution for real-world autonomous systems. Future work may involve relaxing the requirement for known camera intrinsics and integrating more advanced feature-metric losses to further refine predictions.
Summary updated