The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth

Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel J. Brostow, Michael Firman

Abstract

Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the-shelf recurrent networks, which only indirectly make use of the geometric information that is inherently available. We propose ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available. Taking inspiration from multi-view stereo, we propose a deep end-to-end cost volume based approach that is trained using self-supervision only. We present a novel consistency loss that encourages the network to ignore the cost volume when it is deemed unreliable, e.g. in the case of moving objects, and an augmentation scheme to cope with static cameras. Our detailed experiments on both KITTI and Cityscapes show that we outperform all published self-supervised baselines, including those that use single or multiple frames at test time.

Introduction

Estimating depth from a single image is a fundamental challenge in computer vision with significant implications for robotics and augmented reality. While monocular depth estimation has seen rapid progress, it remains inherently limited by the lack of geometric triangulation. Conversely, Multi-View Stereo (MVS) techniques leverage multiple frames to triangulate depth but often rely on expensive ground-truth supervision and struggle in dynamic environments where cameras are static or objects move independently.

ManyDepth addresses this "performance gap" by introducing an opportunistic framework that utilizes multiple video frames at test time when available, while gracefully falling back to single-frame estimation when necessary. Unlike previous methods that rely on slow test-time refinement, ManyDepth operates in a single forward pass, making it highly efficient for real-time applications.

Methodology

ManyDepth builds upon the Monodepth2 framework, extending it with a deep end-to-end cost volume. This structure matches features across consecutive frames to identify the most probable depth through geometric consistency.

The ManyDepth architecture: Source and target frames are processed to create a cost volume, which is then decoded into a depth map.

Adaptive Cost Volumes and Scale

In self-supervised settings, the absolute scale of a scene is typically unknown. ManyDepth overcomes this by learning minimum and maximum depth bounds ($d_{min}, d_{max}$) directly from the data during training. This allows the cost volume to adapt dynamically to the network's internal scale, ensuring robust feature matching across different environments.

Consistency Loss and Motion Masking

A primary failure mode for cost-volume-based methods is the presence of moving objects, which violate the static-world assumption required for triangulation. To mitigate this, ManyDepth employs a monocular "teacher" network to provide a sanity check. A novel consistency loss compares the cost volume's predictions with the monocular network's output. If the predictions diverge significantly—often the case with moving cars or pedestrians—a binary motion mask $M$ is generated, instructing the network to ignore the unreliable cost volume and rely on monocular features instead.

Visualizing the impact of the consistency loss and motion mask M in handling moving objects and low-texture regions.

Handling Static Cameras

MVS-style networks often fail when the camera is stationary (e.g., at a red light) because there is no baseline for triangulation. ManyDepth addresses this through a specific augmentation scheme during training, where the network is occasionally presented with identical frames or zeroed-out cost volumes. This teaches the model to maintain high-quality depth predictions even in the absence of camera motion.

Experimental Results

The model was evaluated on the KITTI and Cityscapes benchmarks, representing standard and complex urban driving scenarios, respectively.

  • State-of-the-Art Performance: ManyDepth outperformed all published self-supervised baselines at the time of publication. On KITTI, it achieved a significant reduction in Absolute Relative (Abs Rel) error, moving from 0.115 (Monodepth2) to 0.098.
  • Qualitative Superiority: The model excels at resolving thin structures and ambiguous, dark regions where single-frame models typically produce blurry artifacts.
  • Efficiency: By utilizing a single forward pass rather than iterative refinement, ManyDepth is approximately 100 times more computationally efficient than competing high-accuracy multi-frame methods.

Conclusion

ManyDepth demonstrates that self-supervised monocular depth estimation can be significantly improved by "opportunistically" leveraging temporal information. By combining the geometric precision of cost volumes with the robustness of monocular recognition through a novel consistency loss, the model provides a scalable and efficient solution for dense depth estimation in real-world, dynamic environments.

Summary updated

Abstract

Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the-shelf recurrent networks, which only indirectly make use of the geometric information that is inherently available. We propose ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available. Taking inspiration from multi-view stereo, we propose a deep end-to-end cost volume based approach that is trained using self-supervision only. We present a novel consistency loss that encourages the network to ignore the cost volume when it is deemed unreliable, e.g. in the case of moving objects, and an augmentation scheme to cope with static cameras. Our detailed experiments on both KITTI and Cityscapes show that we outperform all published self-supervised baselines, including those that use single or multiple frames at test time.
BibTeX
@inproceedings{watson2021temporal,
  author = {Jamie Watson and Oisin Mac Aodha and Victor Prisacariu and Gabriel J. Brostow and Michael Firman},
  title = {{The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=edppjZ8X84w},
}
Figure