Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image

Fangchang Ma, Sertac KaramanICRA 2017 · OpenReview

TL;DR

This paper introduces a novel deep learning approach for predicting dense depth maps from a single RGB image and a sparse set of depth measurements. By fusing RGB data with even a small number of sparse depth samples, their method significantly outperforms purely RGB-based methods, achieving more robust and accurate depth predictions.

What Problem Does This Paper Address?

Estimating accurate and dense depth information from a single RGB image is a notoriously difficult problem in computer vision. Monocular depth estimation is inherently ambiguous and often unreliable, especially in challenging environments or for distant objects. While dedicated depth sensors like LiDAR exist, they are often expensive, provide sparse data, or have limitations like sunlight sensitivity. Existing RGB-based deep learning methods, while improving, still struggle with accuracy, particularly in outdoor scenarios. This paper tackles the challenge of improving depth prediction robustness and accuracy by leveraging the complementary information from sparse depth measurements, which can be readily obtained from low-cost sensors or SLAM systems.

Methods

The core of the proposed method is a single deep regression convolutional neural network (CNN). This network takes both an RGB image and a set of sparse depth samples as input. The authors experimented with various CNN architectures, ultimately settling on a structure similar to state-of-the-art RGB-based methods, featuring a ResNet encoder and a decoder with upsampling layers (specifically, the UpProj module).

Key methodological contributions include:

  • Sparse Depth Integration: The network is designed to directly learn from both RGB and sparse depth data.
  • Depth Sampling Strategy: During training, sparse depth samples are generated "on the fly" by randomly selecting pixels from the ground truth depth map. This strategy is not a standard dropout but aims to increase robustness to varying numbers of input samples and act as a form of data augmentation.
  • Data Augmentation: Standard techniques like random scaling, rotation, color jittering, and flipping are applied to the training data to improve generalization.
  • Loss Function: While exploring L2 and Reversed Huber (berHu) losses, the authors found that the L1 loss function yielded slightly better results and was chosen as the default for its simplicity and performance.
  • Architecture Choices: The paper details experiments on different upsampling layers and the initial convolution layer, demonstrating the effectiveness of the UpProj module and a standard convolution, respectively.

Datasets

The proposed method was trained and evaluated on two widely used datasets:

  • NYU-Depth-v2: An indoor dataset consisting of RGB and depth images captured by a Microsoft Kinect. The authors used the official split for training and testing, with a specific subset of the labeled test data for benchmarking.
  • KITTI Odometry Dataset: An outdoor dataset containing synchronized camera and LiDAR measurements. The authors utilized a portion of the odometry sequences for training and a subset of the test sequences for evaluation. Notably, they highlight that the ground truth depth for KITTI is sparser than for NYU-Depth-v2, making it a more challenging benchmark.

Key Insights and Ideas

  • Sparse Depth is Powerful: Even a small number of spatially random sparse depth samples (e.g., 100) can dramatically improve depth prediction accuracy compared to using RGB alone. This suggests that sparse depth provides crucial geometric cues that RGB alone struggles to infer reliably.
  • Complementary Information: The fusion of RGB and sparse depth data is more than just additive; the network learns a suitable representation that leverages the strengths of both modalities. Color information helps, especially when the number of sparse samples is low.
  • Robustness to Sample Count: The proposed sampling strategy during training helps the network become robust to variations in the number of available sparse depth samples.
  • Beyond RGB: The paper convincingly argues that relying solely on RGB for dense depth prediction has fundamental limitations, especially in outdoor environments, and that incorporating sparse geometric data is a promising direction.
  • Practical Applications: The method is not just an academic exercise; it's demonstrated to be effective as a plug-in for existing SLAM systems to create dense maps and for super-resolution of sparse LiDAR data.

Key Results

  • NYU-Depth-v2: With just 100 sparse depth samples, the RMSE was reduced by over 50% compared to RGB-only prediction (from 51cm to ~25cm). The percentage of reliable predictions (δ₁) also saw a significant boost.
  • KITTI: On this more challenging outdoor dataset, 100 sparse depth samples reduced the RMSE by half (from ~7m to ~3.5m) and increased the percentage of reliable predictions from 59% to 92%.
  • Comparison to State-of-the-Art: The proposed RGBd method, even with fewer sparse samples, outperforms previous fusion-based methods (e.g., [24] with 225 laser measurements) and significantly surpasses RGB-only methods on both datasets.
  • LiDAR Super-Resolution: The method achieved an average REL of 4.9% for LiDAR super-resolution, a substantial improvement over RGB-only prediction (20.8%), making distant objects more recognizable.

Limitations and Future Work

  • Blurry Boundaries: The authors acknowledge that their current bottleneck network architecture can lead to blurry boundaries in the predicted depth maps, even with many depth samples. They hypothesize that incorporating skip connections from encoders to decoders might help improve performance in this regard.
  • Trusted Region for SLAM: In the SLAM application, the method preserves only pixels within a "trusted region" defined by the convex hull of input sparse depth samples. Pixels outside this region (e.g., on featureless walls) are removed, leading to incomplete dense maps in certain scenarios.
  • Future Directions: The paper suggests exploring the impact of injecting random noise and alternative sampling strategies (e.g., feature points) for depth sampling. Further investigation into network architectures, particularly those with skip connections, is also implied for addressing the boundary issue.

Summary updated

Abstract

We consider the problem of dense depth prediction from a sparse set of depth measurements and a single RGB image. Since depth estimation from monocular images alone is inherently ambiguous and unreliable, to attain a higher level of robustness and accuracy, we introduce additional sparse depth samples, which are either acquired with a low-resolution depth sensor or computed via visual Simultaneous Localization and Mapping (SLAM) algorithms. We propose the use of a single deep regression network to learn directly from the RGB-D raw data, and explore the impact of number of depth samples on prediction accuracy. Our experiments show that, compared to using only RGB images, the addition of 100 spatially random depth samples reduces the prediction root-mean-square error by 50% on the NYU-Depth-v2 indoor dataset. It also boosts the percentage of reliable prediction from 59% to 92% on the KITTI dataset. We demonstrate two applications of the proposed algorithm: a plug-in module in SLAM to convert sparse maps to dense maps, and super-resolution for LiDARs. Software and video demonstration are publicly available.
BibTeX
@inproceedings{ma2017sparse,
  author = {Fangchang Ma and Sertac Karaman},
  title = {{Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2017},
  url = {https://openreview.net/forum?id=YW47bVB2bV8},
}
Figure