Sparse-to-Continuous: Enhancing Monocular Depth Estimation using Occupancy Maps

Nícolas dos Santos Rosa, Vitor Guizilini, Valdir Grassi Jr.

TL;DR

This paper tackles the challenge of monocular depth estimation (SIDE) by proposing a novel "Sparse-to-Continuous" approach. They leverage Hilbert Maps to densify sparse LiDAR depth data into continuous occupancy maps, which are then used to train deep neural networks, significantly improving depth prediction accuracy without requiring additional sensors or complex network architectures.

What Problem Does This Paper Address?

Estimating depth from a single image (SIDE) is an inherently ill-posed problem because a 2D image can correspond to multiple 3D scenes. While deep learning has made significant strides, a major bottleneck is the quality of training data. Supervised methods often rely on ground-truth depth maps, but these are typically sparse, especially from sensors like LiDAR, leaving large portions of the scene without depth information. This sparsity forces networks to either learn from incomplete data or requires the use of synthetic datasets, which can suffer from a realism gap. The paper aims to improve the quality of monocular depth estimation by addressing the sparsity of training data.

Methods

The core innovation is a "densification" method using the Hilbert Maps framework. Instead of directly using sparse LiDAR point clouds, the authors first create a continuous occupancy map from the 3D points. This continuous map can then be projected into a 2D depth map at any desired resolution. This process effectively "fills in" the missing depth information in a spatially consistent manner.

The densified, continuous depth maps are then used as supervision for training a Fully Convolutional Residual Network (FCRN), inspired by ResNet-50. This network architecture is chosen for its efficiency and ability to handle residual learning for improved accuracy. The authors also employ standard data augmentation techniques like horizontal flips and color distortion, carefully avoiding scaling and translation to preserve world-space geometry. For training, they experiment with three common loss functions: Mean Squared Error (MSE), Eigen's Squared Relative Error, and the Adaptive BerHu Penalty, finding BerHu to be most effective.

Datasets

The primary dataset used is KITTI. The authors differentiate between:

  • KITTI Depth (Semi-dense): This is the standard KITTI depth dataset, characterized by sparse depth maps derived from LiDAR scans.
  • KITTI Continuous (Dense): This is a new benchmark created by the authors. They generate dense depth maps by applying their Hilbert Maps densification process to the KITTI Raw data. This dataset provides a continuous occupancy map as supervision, effectively filling in the sparse regions of the original KITTI Depth.

Key Insights and Ideas

  • Densification is Key: The paper strongly suggests that improving the quality of the training data (making it denser and more continuous) is more impactful than solely relying on more complex network architectures or additional sensors.
  • Hilbert Maps for Densification: The Hilbert Maps framework proves to be an effective tool for creating continuous occupancy maps from sparse 3D data, which can then be used for depth map densification.
  • Real-World Sparsity is a Major Hurdle: The KITTI Depth dataset, despite being a standard, has a significant percentage of invalid pixels (e.g., for sky or distant objects), which poses a challenge for training robust depth estimation models.
  • Continuous Data Outperforms Sparse Data: Training with the densified "KITTI Continuous" dataset consistently yields better results than training with the original sparse "KITTI Depth" dataset, even when using fewer training samples.
  • Geometric Consistency is Preserved: The Hilbert Maps approach, by modeling occupancy, helps maintain spatial dependencies and provides a margin of safety around obstacles, unlike some other methods.

Key Results

The paper demonstrates significant improvements in depth estimation accuracy when training with their proposed "Sparse-to-Continuous" method.

  • Quantitative Improvements: When evaluated on the Eigen Split of the KITTI Depth dataset, models trained on the "KITTI Continuous" dataset consistently outperformed those trained on the sparse "KITTI Depth" dataset across various metrics (Abs Rel, Sqr Rel, RMSE, RMSE (log), and thresholded accuracy). For instance, the "Ours (KITTI Continuous, dense, berhu)" model achieved an Abs Rel of 0.119 and an RMSE (log) of 0.180 on the 0-50m range, outperforming the sparse counterpart.
  • Efficiency Gain: The "KITTI Continuous" dataset required 67.6% fewer training examples than the "KITTI Depth" dataset to achieve superior performance, highlighting the efficiency of using denser supervision.
  • Qualitative Improvements: Figures 3 and 4 show that the depth predictions from models trained on continuous data are significantly less blurred, have sharper edges, and are more accurate, especially in areas that were previously sparse.

Limitations and Future Work

  • Computational Cost: The authors acknowledge that the preprocessing step of computing continuous depth maps using Hilbert Maps is computationally intensive.
  • Reconstructive Capabilities: The effectiveness of the Hilbert Maps framework is bounded by its inherent reconstructive capabilities.
  • Future Work: The authors plan to optimize the Hilbert Maps preprocessing method, explore new architectural developments for SIDE, investigate other depth-only in-painting techniques for comparison, and analyze the impact of color distortion on depth estimates.

Summary updated

Abstract

This paper addresses the problem of single image depth estimation (SIDE), focusing on improving the quality of deep neural network predictions. In a supervised learning scenario, the quality of predictions is intrinsically related to the training labels, which guide the optimization process. For indoor scenes, structured-light-based depth sensors (e.g. Kinect) are able to provide dense, albeit short-range, depth maps. On the other hand, for outdoor scenes, LiDARs are considered the standard sensor, which comparatively provides much sparser measurements, especially in areas further away. Rather than modifying the neural network architecture to deal with sparse depth maps, this article introduces a novel densification method for depth maps, using the Hilbert Maps framework. A continuous occupancy map is produced based on 3D points from LiDAR scans, and the resulting reconstructed surface is projected into a 2D depth map with arbitrary resolution. Experiments conducted with various subsets of the KITTI dataset show a significant improvement produced by the proposed Sparse-to-Continuous technique, without the introduction of extra information into the training stage.
BibTeX
@inproceedings{rosa2019sparse,
  author = {Nícolas dos Santos Rosa and Vitor Guizilini and Valdir Grassi Jr.},
  title = {{Sparse-to-Continuous: Enhancing Monocular Depth Estimation using Occupancy Maps}},
  booktitle = {Icar},
  year = {2019},
  doi = {10.1109/icar46387.2019.8981652},
}
Figure