The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth

Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel J. Brostow, Michael Firman

Abstract

Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the-shelf recurrent networks, which only indirectly make use of the geometric information that is inherently available. We propose ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available. Taking inspiration from multi-view stereo, we propose a deep end-to-end cost volume based approach that is trained using self-supervision only. We present a novel consistency loss that encourages the network to ignore the cost volume when it is deemed unreliable, e.g. in the case of moving objects, and an augmentation scheme to cope with static cameras. Our detailed experiments on both KITTI and Cityscapes show that we outperform all published self-supervised baselines, including those that use single or multiple frames at test time.

Introduction

Predicting 3D depth from a single 2D image is a fundamental challenge for autonomous driving and augmented reality. While self-supervised methods have advanced significantly by learning from video sequences, they typically discard temporal context at inference time, relying solely on a single frame. ManyDepth addresses this inefficiency by "opportunistically" utilizing multiple frames at test time to enhance depth accuracy through a geometric cost-volume approach.

Methodology

ManyDepth builds upon a plane-sweep stereo architecture, introducing three innovations to adapt it for self-supervised monocular video:

Adaptive Cost Volumes

Standard Multi-View Stereo (MVS) requires predefined depth ranges. Since self-supervised models learn depth only up to an unknown scale, ManyDepth dynamically learns depth extents ($d_{min}$ and $d_{max}$) from the data during training using an exponential moving average of its own predictions.

Consistency Loss

MVS assumes a static world, but real-world driving sequences are full of moving objects. ManyDepth uses a separate monocular network as a "teacher" during training. If the cost volume's geometric matching disagrees with the monocular prediction, a consistency mask is triggered, forcing the model to ignore the unreliable cost volume and rely on monocular semantic cues instead.

ManyDepth produces sharp depth maps for thin structures like signposts by effectively utilizing temporal information.

Static Camera Augmentation

Geometric triangulation fails when the camera is stationary. To ensure robustness, the authors augment training by randomly replacing previous frames with static or color-augmented versions of the current frame, teaching the network to fall back on monocular cues when motion is absent.

Results and Discussion

ManyDepth outperforms all published self-supervised baselines on the KITTI and Cityscapes benchmarks. On the KITTI Eigen test set, it achieves an Absolute Relative Error (Abs Rel) of 0.087.

Without proper consistency mechanisms, multi-frame models often suffer from blurring and artifacts (red circle) in complex scenes.

Beyond accuracy, ManyDepth is highly efficient. Unlike methods requiring expensive test-time optimization, it uses a single forward pass, making it roughly 100x faster and suitable for real-time applications.

Conclusion and Future Work

By intelligently combining geometric MVS with monocular semantic reasoning, ManyDepth provides a robust framework for multi-frame depth estimation. While it remains dependent on accurate pose estimation and fixed camera intrinsics, its ability to switch between geometric and semantic cues makes it a robust solution for real-world autonomous systems. Future work may involve relaxing the requirement for known camera intrinsics and integrating more advanced feature-metric losses to further refine predictions.

Summary updated

Abstract

Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the-shelf recurrent networks, which only indirectly make use of the geometric information that is inherently available. We propose ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available. Taking inspiration from multi-view stereo, we propose a deep end-to-end cost volume based approach that is trained using self-supervision only. We present a novel consistency loss that encourages the network to ignore the cost volume when it is deemed unreliable, e.g. in the case of moving objects, and an augmentation scheme to cope with static cameras. Our detailed experiments on both KITTI and Cityscapes show that we outperform all published self-supervised baselines, including those that use single or multiple frames at test time.
BibTeX
@inproceedings{watson2021temporal,
  author = {Jamie Watson and Oisin Mac Aodha and Victor Prisacariu and Gabriel J. Brostow and Michael Firman},
  title = {{The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=edppjZ8X84w},
}
Figure