Deep Learning based Monocular Depth Prediction: Datasets, Methods and Applications

Qing Li, Jiasong Zhu, Jun Liu, Rui Cao, Sen Jia, Guoping Qiu

Abstract

Estimating depth from RGB images can facilitate many computer vision tasks, such as indoor localization, height estimation, and simultaneous localization and mapping (SLAM). Recently, monocular depth estimation has obtained great progress owing to the rapid development of deep learning techniques. They surpass traditional machine learning-based methods by a large margin in terms of accuracy and speed. Despite the rapid progress in this topic, there are lacking of a comprehensive review, which is needed to summarize the current progress and provide the future directions. In this survey, we first introduce the datasets for depth estimation, and then give a comprehensive introduction of the methods from three perspectives: supervised learning-based methods, unsupervised learning-based methods, and sparse samples guidance-based methods. In addition, downstream applications that benefit from the progress have also been illustrated. Finally, we point out the future directions and conclude the paper.

Introduction

In computer vision, monocular depth estimation is a classic "ill-posed" problem. Mathematically, an infinite number of 3D scenes can project onto the same 2D image. Humans solve this using context—such as knowing the typical size of a car—but for machines, this is incredibly difficult. While LiDAR and stereo cameras provide direct depth data, they are often expensive, power-hungry, or bulky. If we can teach a single standard camera to "see" depth accurately, we unlock massive potential for indoor navigation, autonomous driving, and augmented reality on consumer devices. This paper surveys the transition from traditional, slow machine learning methods to the current state-of-the-art deep learning architectures.

Methodologies for Depth Prediction

The field is categorized into three primary pillars based on the learning strategy and data requirements:

Supervised Learning

These models are trained on pairs of RGB images and ground-truth depth maps. They generally fall into two camps: Regression-based (predicting a continuous depth value) and Classification-based (discretizing depth into "bins"). A major trend here is Multi-task Learning, where the network simultaneously predicts depth, surface normals, and semantic labels, allowing the tasks to regularize and improve each other.

Typical encoder-decoder architecture for monocular depth estimation, mapping an input RGB image to a dense depth map.

Unsupervised Learning

To bypass the need for expensive ground-truth data, these methods use geometry as a supervisor. By using Stereo Pairs (warping one view to another) or Video Sequences (predicting both depth and camera ego-motion), the model learns depth by minimizing "photometric error"—essentially checking if its 3D guess allows it to accurately reconstruct one image from another.

Sparse Samples Guidance

This "hybrid" approach uses a small number of real depth points (e.g., from a low-resolution LiDAR) to guide the RGB-based prediction. The paper discusses Early Fusion (stacking RGB and sparse depth at the input) versus Late Fusion (processing them in separate branches and merging features later).

Datasets and Benchmarks

The survey covers the essential benchmarks that have driven the field forward:

  • Indoor: NYU Depth V2 is the gold standard, featuring 464 scenes captured with a Microsoft Kinect.
  • Outdoor: KITTI is the most famous, providing LiDAR-based ground truth for autonomous driving scenarios.
  • Hybrid/General: Make3D offers a mix of environments, while newer datasets like MegaDepth use Structure-from-Motion (SfM) on internet photos to create massive training sets.
  • Remote Sensing: The paper uniquely highlights ISPRS and GRSS datasets, which are used for height estimation from aerial imagery.

Performance and Key Results

Supervised methods currently lead the pack in terms of raw accuracy. Top-performing models on the NYU dataset, such as Jiao et al., achieve a Mean Relative Error (REL) as low as 0.098, representing a massive improvement over early deep models.

Comparison of state-of-the-art monocular depth estimation methods across various metrics and strategies, highlighting the performance gap between regression, classification, and sparse-guided approaches.

When even a tiny amount of LiDAR data (sparse samples) is added, the Root Mean Square Error (RMSE) on the KITTI dataset can drop from approximately 3.0 meters to under 0.8 meters, demonstrating that "hints" are incredibly effective. Furthermore, the emergence of lightweight architectures like FastDepth has enabled real-time performance at 30+ FPS on embedded devices.

Future Directions

Despite significant progress, several "frontier" challenges remain:

  • Scale Ambiguity: Unsupervised models still struggle to determine absolute distance without external scaling factors.
  • Generalization: Domain adaptation remains a major hurdle; models trained in one environment often fail in others with different weather or lighting.
  • Explainability: Understanding the internal logic of CNNs for depth estimation—whether they rely on vertical position, texture, or object type—is still an open question.
  • Dynamic Objects: Most unsupervised models assume a static world. Moving objects break geometric training assumptions, leading to artifacts like "infinite depth."

Summary updated

Abstract

Estimating depth from RGB images can facilitate many computer vision tasks, such as indoor localization, height estimation, and simultaneous localization and mapping (SLAM). Recently, monocular depth estimation has obtained great progress owing to the rapid development of deep learning techniques. They surpass traditional machine learning-based methods by a large margin in terms of accuracy and speed. Despite the rapid progress in this topic, there are lacking of a comprehensive review, which is needed to summarize the current progress and provide the future directions. In this survey, we first introduce the datasets for depth estimation, and then give a comprehensive introduction of the methods from three perspectives: supervised learning-based methods, unsupervised learning-based methods, and sparse samples guidance-based methods. In addition, downstream applications that benefit from the progress have also been illustrated. Finally, we point out the future directions and conclude the paper.
BibTeX
@misc{li2020deep,
  author = {Qing Li and Jiasong Zhu and Jun Liu and Rui Cao and Qingquan Li and Sen Jia and Guoping Qiu},
  title = {{Deep Learning based Monocular Depth Prediction: Datasets, Methods and Applications}},
  year = {2020},
  eprint = {2011.04123},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.04123},
}
Figure